RESEARCH/Original article
Quality assessment of X-rays interpreted via teleradiology for Me´decins Sans Frontie`res
Journal of Telemedicine and Telecare 2014, Vol. 20(2) 82–88 ! The Author(s) 2014 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/1357633X14524153 jtt.sagepub.com
Saskia Spijker1, Savvas Andronikou2,3,4, Cara Kosack1, Richard Wootton5,6, Maryline Bonnet7 and Nathalie Lemmens8
Summary Me´decins Sans Frontie`res (MSF) is a humanitarian organisation which provides emergency medical aid in challenging settings; field staff often diagnose and treat patients using limited resources and without the expertise of specialists. Teleradiology is available for MSF sites which use digital computed radiography (CR) imaging or conventional film and chemistry. We conducted a retrospective study of the quality of X-rays utilised by MSF for teleradiology diagnosis over a one-year period. All plain X-ray examinations referred for interpretation using two MSF teleradiology platforms in 2012 were assessed against 15 image criteria and further evaluated as being either diagnostic or non-diagnostic. The sites studied sent an average of 115 images (range 10452). Images were a mixture of chest, skeletal and abdominal radiographs. The majority of the images were CR (n ¼ 597, 74%). Three sites were MSF/Epicentre installed and operated (Epicentre is a research facility affiliated with MSF); five sites were operated by the ministry of health, imaging patients referred by MSF. The sites performing poorest for quality were all facilities which used film and chemistry (53% non-diagnostic images). The sites performing better for quality were facilities which used CR digital imaging (12% non-diagnostic images), two of which had also undergone radiographer training. Our study suggests that transitioning to CR digital imaging has the potential to improve image quality compared to film and chemistry. Radiography training should be made a priority for all sites with X-ray services. The continued utilisation of X-ray services by MSF where images have proven to be consistently poor should be re-considered. Accepted: 17 December 2013
Introduction Me´decins Sans Frontie`res (MSF) is a non-governmental humanitarian organisation that responds to emergency situations and provides medical assistance to those in need. MSF teams provide medical emergency aid in difficult settings and staff often have to diagnose and treat patients with inadequate resources.1 Human resource shortages are common and the expertise of specialists such as radiologists is often lacking. Radiology is available in many locations where MSF provides medical services and teleradiology is available at all sites that have an Internet connection. Teleradiology is available both for sites that use digital computed radiography (CR) imaging and for those using film and chemistry followed by digitisation with a digital camera. MSF utilises two platforms for teleradiology: MSF’s own general telemedicine system (based on the Collegium Telemedicus2) and vRad (Virtual Radiologic Corporation, Eden Prairie, MN, USA), a commercial teleradiology company offering pro-bono services to MSF. Some of the referring X-ray departments are MSF installed and operated, while others are ministry of health (MOH) facilities within a hospital or referral centre. In all cases the purpose of teleradiology is to
obtain assistance in reaching a diagnosis and deciding the best clinical management of the patient. Quality assurance (QA) in radiology is an important component in ensuring high image quality.3 Evaluating image quality can identify and address the causes of poor quality images.4 These may be due to limitations in the competence of the radiographers (X-ray technicians), equipment problems, or a combination of both.5 QA in teleradiology should always include the quality of the radiographic image. This is particularly important in
1
Me´decins Sans Frontie`res International, Amsterdam, The Netherlands Department of Radiology, Faculty of Health Sciences, University of the Witwatersrand, South Africa 3 World Federation of Paediatric Imaging, Reston, USA 4 South African Medical Unit, Me´decins Sans Frontie`res, South Africa 5 Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, Tromsø, Norway 6 Faculty of Health Sciences, University of Tromsø, Norway 7 Epicentre, Geneva, Switzerland 8 Onze Lieve Vrouwe Gasthuis, Amsterdam, The Netherlands 2
Corresponding author: Saskia Spijker, Plantage Middenlaan 14, 1018 DD, Amsterdam, The Netherlands. Email: [email protected]
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
Spijker et al.
83
resource-limited settings where there are greater challenges to achieving high quality images. The aim of the present study was to assess the images sent for teleradiology from routine MSF field sites.
Methods We conducted a retrospective study to determine the quality of X-rays utilised by MSF for teleradiology diagnosis over a one-year period. All plain X-ray examinations referred for interpretation using MSF teleradiology platforms in 2012 were assessed. Sites which sent fewer than 10 X-rays for teleradiology consultation in 2012 were excluded from the analysis. At sites using CR, X-ray images were produced in the medical imaging file format DICOM (Digital Imaging and Communications in Medicine) and were transmitted for teleradiology in this format. All the sites with CR imaging had Carestream CR readers. At sites using film and chemistry, hard copy X-ray films were digitised to JPEG files using a digital camera before transmission (JPEG is a commonly used method of lossy compression for digital photography). Sites using this method used their own digital camera and were encouraged to follow a protocol developed by MSF, based on previous studies about digitising X-ray films.6–8 The protocol suggested an image size of 3.5–5 Mpixel, high compression setting and exposure compensation setting of þ1.3 EV. All X-ray images were transmitted anonymously with a clear patient history. Images sent to vRad were transmitted via a secure Virtual Private Network (VPN) and cases transmitted through the MSF Collegium Telemedicus system were sent via Secure Messaging.
Quality assessment The initial quality assessment was performed by Observer 1 (a radiographer with more than 15 years experience). All images were viewed on a flat-screen monitor (MultiSync LCD 3090WQXi, NEC) at approximately 350 cd/m2 brightness. Each image was evaluated against 15 criteria. The criteria fell into six categories: positioning (1. rotation, 2. angulation, 3. other); collimation (4. limited anatomy, 5. excessive anatomy); exposure (6. contrast too high, 7. contrast too low, 8. density too high, 9. density too low); artefacts (10. motion, 11. other); markers (12. incorrect, 13. misplaced, 14. absent) or other (15. others). An image could be marked as being unacceptable in more than one criterion in a category. For example, a chest X-ray could be marked twice in the exposure group if it was evaluated as being too low in density and also too low in contrast, or twice for positioning if it was both rotated and with poor inspiration. In addition to being considered against these criteria, each image was evaluated as being of diagnostic or nondiagnostic quality overall. In view of the subjective nature of assessing quality in X-ray images, a quality reference guide was developed and used to determine the point at which an image would become non-diagnostic.
For example, a lateral wrist X-ray could be slightly rotated, have excessive collimation and be missing a marker, but could still be considered as a diagnostic image overall. On the other hand, a PA chest X-ray that was too high in density (too dark) but otherwise technically perfect would be considered non-diagnostic because this one factor limited its ability to be interpreted. Under normal circumstances, all X-ray images should be clearly marked with a unique patient identification, either name or file number, and with the patient’s date of birth, date of the X-ray and clinic/hospital identifier.9 This is a minimum legal requirement, but it was not evaluated in the present study because all MSF teleradiology images are sent without patient identifiers in accordance with the organisation’s privacy policy.
Data analysis We calculated the overall proportion and proportion per site of non-diagnostic film images and non-diagnostic CR digital images with corresponding 95% confidence intervals (CIs). These combined proportions were compared using the chi-squared test. The proportion of film and CR digital images requiring improvement was calculated for each group of quality assessment criteria. A random sample of the X-ray images was read independently by a second observer (a radiographer also with more than 15 years experience) using the same monitor as observer 1, in order to determine the agreement in the overall quality assessment between the two observers. The sample size was determined by assuming an expected kappa of 0.7, with a proportion of 25% of non-diagnostic X-rays by observer 1, 20% of discordance between the two observers, 10% precision and a 95% CI. This gave a minimum sample size of 245 images. Weighted kappa values were calculated with their 95% CIs and were interpreted as follows: kappa < 0 was considered to indicate no agreement; kappa ¼ 0.0 to 0.20 as slight agreement; kappa ¼ 0.21 to 0.40 as fair agreement; kappa ¼ 0.41 to 0.60 as moderate agreement; kappa ¼ 0.61 to 0.80 as substantial agreement; kappa ¼ 0.81 to 1.00 as almost perfect agreement.10 Data were analysed using a standard package (Stata 12.1, College Station, Texas, USA).
Results During the one-year study period, a total of 818 plain Xray images were sent for teleradiology consultation from 14 sites. Of the 14 sites, seven sites sent fewer than 10 images each and these sites were excluded from further analysis. Thus, the final sample concerned 806 images. The seven included sites sent an average of 115 images (range 10-452). Images were a mixture of chest, skeletal and abdominal radiographs, see Table 1. The majority of the images were CR: 74% (n ¼ 597) were CR and 26% (n ¼ 209) were film. The 806 images were from 623 individual patients; 64% (n ¼ 397) were adults and 36% (n ¼ 226) were children aged 15 years or under.
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
84
Journal of Telemedicine and Telecare 20(2)
Table 1. Sites sending at least 10 X-ray cases for teleradiology in 2012. Note that one site (Central Asia 2) sent both film and CR images – these are listed separately. No. of images
No. of non-diagnostic images
Percentage of non-diagnostic images
95% confidence interval
MOH MOH MOH MOH MOH
23 44 89 43 10 209
20 25 47 17 2 111
87 57 53 40 20 53
68–96 41–70 42–63 25–54 3–51 46–59
Epicentrea MSFb MSF
128 17 452 597
19 2 48 69
15 12 11 12
9–21 2–33 8–13 9–14
Facility Film Central Asia 1 Sub-Saharan Africa Central Asia 2 Sub-Saharan Africa Southeast Asia 1 Total CR Sub-Saharan Africa Central Asia 2 Sub-Saharan Africa Total
1 2
3 4
a
Research facility affiliated with MSF, utilising the MOH radiology department. Facility installed and staff trained by MSF, subsequently handed over to MOH.
b
Figure 1. Proportion of non-diagnostic images at sites using film and CR.
Primary outcome measure
Individual image criteria
The sites using film and chemistry for X-ray imaging had significantly more non-diagnostic images (53%, 95% CI 46-60) than sites with CR imaging (12%, 95% CI 9–14), see Table 1. This difference was significant (P < 0.001). Sites using film and chemistry tended to submit fewer cases for teleradiology, see Figure 1. Sites using CR tended to be those operated directly by MSF, see Figure 2. There was good agreement between the two observers in their assessments of film images (kappa ¼ 0.67, 95% CI 0.51–0.83) and almost perfect agreement for CR images (kappa ¼ 0.91, 95% CI 0.79–1.00).
Of the 209 film images, a total of 366 criteria were marked as requiring improvement. Of these images, the criteria marked most frequently were in the exposure category (68%, n ¼ 143), see Table 2. Of the 597 CR images, a total of 311 criteria were marked as requiring improvement. The criteria marked most frequently were in the collimation category (19%, n ¼ 116). The results of the individual criteria evaluated for improvement are shown in Figure 3. Collimation was considered an area for improvement in 25% (n ¼ 53) of the film images with limited
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
Spijker et al.
85
Figure 2. Proportion of non-diagnostic images at MOH and MSF sites. The site marked with an asterisk was originally installed and the staff trained by MSF, but was subsequently handed over to the MOH.
anatomy coverage being more common than excessive anatomy coverage (23% and 2% respectively). CR images were slightly better for collimation (19%, n ¼ 116) demonstrating almost equal limited and excessive anatomy coverage (9% and 10% respectively). In all instances of film and CR images with limited anatomy coverage, the radiographers had not recognised that the anatomy on the acquired X-ray was incomplete and had failed to repeat the film to ensure that all anatomy was covered. Inadequate exposure was the reason most frequently evaluated on film images as requiring improvement with too high contrast and too high density being the most common individual reasons. Both film and CR images are equally vulnerable to motion artefact and this was demonstrated on 6% (n ¼ 12) of the film images and 4% (n ¼ 24) of the CR images. Poor processing technique was included in the ‘artefacts-other’ category and this was the greatest contributor to this category. 24% (n ¼ 51) of film images were assessed as demonstrating artefacts in the ‘artefacts-other’ category compared with 1% (n ¼ 7) of CR images. In CR, processing artefacts rarely occur because there is no film and chemistry component. Absent, misplaced or incorrect markers occurred in 42% (n ¼ 89) of film images. CR images were marked against the marker criteria group on 6% of the images (n ¼ 40); 26 of these images were missing a side marker and the opportunity to add the marker at a postprocessing step was missed. The absence of a side marker would not necessarily cause a radiograph to be evaluated as non-diagnostic. However, use of side markers represents meticulous technique that is critical for both patient care and for medico-legal protection.
Discussion Quality assurance in radiography involves QA of the diagnostic X-ray equipment and review of the quality of the radiographs produced.11,12 Even though no single individual group or scientific/professional society contains all of the skills or knowledge required to perform QA efficiently11 it is important that all hospitals or clinics using radiography perform some kind of QA within their available resources. Good radiographic technique is the main factor in improving image quality.13 Experienced and dedicated staff are a key factor, but in the context of humanitarian aid agencies working in resource-limited settings, this can be a challenge. Wherever X-rays are taken, QA of radiographic quality is feasible and can be performed either by the people taking the radiographs (the radiographer) or by those interpreting them (the radiologist). One of the primary responsibilities of the radiographer is to ensure an adequate level of image quality before submitting the image to the radiologist for interpretation. In radiology departments, QA is generally performed by radiographers or radiologists undertaking a film reject analysis. The reject film rate is a good quality indicator.14 While there is no universally-agreed cut-off for an acceptable reject film rate,15 some advise that the overall rate should remain below 10%.16 Caution should be given, however, to using the reject rate as the sole representation of the quality of a radiology department because it is sensitive to examination type.14 The role of such an analysis should be to isolate specific quality problems that result in images that need to be rejected or repeated.14 Rejection rate is difficult to determine in teleradiology where the rejected hard copy films or CR images cannot be accessed, i.e. other images may have been taken and then discarded as
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
Journal of Telemedicine and Telecare 20(2) inadequate before the teleradiology transmission. On the other hand, the lack of experienced or supervised radiography staff may result in inadequate images being accepted and sent for teleradiology interpretation without any repetition. Although our quality assessment of images sent for teleradiology was not a film reject analysis, it employed the same principles.
87 42
26 4
1 –
10 2
4 2 2 0 – – Absent Misplaced
7 1 26 4 6 1 30 5
3 1
6 1
4 2
8 1
Film Percentage of 209 images Total group % CR Percentage of 597 images Total group %
7 3 7 48 8 10
Other Angulation Rotation
*each image could be marked against none or several criteria or twice within one category.
60 10
5 2
57 27 68 0 – 10
18 9
51 24
17 8
12 6 30 24 4 5
51 24
1 – 42 4 – 6
Film vs. CR
48 23 25 56 9 19
Incorrect Motion Contrast too high Limited anatomy
Excessive anatomy
Exposure Collimation Positioning
Table 2. Criteria for improvement in 806 plain X-ray images (film ¼ 209; CR ¼ 597).*
Contrast too low
Density too high
Density too low
Artefacts
Other
Markers
Other
86
The five sites performing the worst for quality (>20% nondiagnostic) were all MOH facilities using film and chemistry. The most common reasons for poor quality were inadequate exposure and artefacts due to processing. These areas all directly relate to the difficulty in obtaining good image quality using film and chemistry, where far greater technical knowledge and experience is required of radiographers in exposure selection and processing technique. Successful film processing also relies on a clean water supply and stable electricity to regulate the water temperature in the water bath surrounding the chemistry. Both of these factors are challenging in resource-limited settings. The three sites performing better for quality (15% or less non-diagnostic) were using CR digital imaging and were MSF/Epicentre sites. While the image quality was significantly higher at those sites, the percentage of non-diagnostic images still leaves room for improvement. There were far fewer problems with exposure due to the greater latitude within a digital system to compensate for exposure errors and the complete avoidance of contrast/density and artefact faults encountered with chemical processing of film. The images that were evaluated as non-diagnostic by the sites with CR were predominantly due to collimation (both limited and excessive demonstration of anatomy), positioning (particularly rotation of anatomy) and post-processing (not setting best contrast/density digitally). These factors all reflect the technique of the radiographers and support the need for continued training of radiographers in how to identify a high quality X-ray image. The results of the study show that the departments which sent more images for teleradiology also produced better quality images. However, we do not know what proportion of each department’s workload was sent for teleradiology.
Radiographer training The three sites with CR that performed best for quality were sites where teleradiology services were planned as a part of the X-ray installation. Two of these sites (SubSaharan Africa 4 and Central Asia 2 with CR) had also received training by an MSF expatriate radiographer (at the time of the X-ray machine and CR installation) and this probably contributed to both sites reporting the lowest percentages of non-diagnostic images. Staff at the site Sub-Saharan Africa 3 had also received technical training on the CR digital reader and teleradiology transmission, but this training did not extend to radiographic
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
Spijker et al.
87
Figure 3. Criteria for improvement. Each image could be marked against none or several criteria (see text).
technique inside the X-ray room itself. CR digital imaging is likely to contribute to a higher level of image quality but with extended radiographer training this could be improved even further. Where MSF utilise MOH X-ray facilities, this is usually done on a referral basis without the possibility of influencing equipment choices or the possibility for conducting training. This probably contributed to the significant difference in image quality between MSF sites with digital imaging where training had taken place and MOH sites using film and chemistry where no training had taken place.
been followed and the digitised image evaluated for quality was probably worse than the original film. This is less of a problem for images considered non-diagnostic due to limited anatomy or excessive artefacts, which are not influenced by camera digitisation. Our study, however, was evaluating the end result, i.e. the image sent to the radiologist for interpretation on the teleradiology platform. Investigating the importance of good camera technique in the digitisation process is an area for potential future investigation if conversion to CR is not possible.
Conclusion Limitations The present study had certain limitations. For example, image quality assessment was performed by a single observer and might therefore have been biased. However, an independent double-reading of a sample of the images by a second observer demonstrated substantial agreement for film images and almost perfect agreement for CR images. Nonetheless, evaluating an image is a subjective process in the absence of a gold standard. The difference in the kappa values observed between film and CR images could be explained by the smaller sample of film images. It was not always clear when viewing digitised film images (JPEG format) if the quality reflected the true quality of the original X-ray film or the influence of the digitisation process itself. Lung markings visible on an X-ray film may be less visible once digitised if the exposure value of the camera was set incorrectly, the flash was used, or if extraneous light around the X-ray was included in the field of view. It is difficult to know which sites strictly adhered to the protocol provided on how to digitise an X-ray film. In some cases it was very obvious that the protocol had not
The multi-country data that MSF has access to through its routine operations demonstrated a clear difference in quality between film and digital CR images. Radiographer training that was given to the MSF sites with CR imaging probably contributed to this difference in image quality. The following recommendations for MSF and other users of X-ray in resource-limited settings can be made: 1. Transition to CR imaging. Our assessment suggests that adopting CR imaging has the potential to improve image quality compared to film and chemistry, due to the technology compensating somewhat for radiographic variables (exposure, film processing procedures). This is a major advantage in resourcelimited settings. Sites where X-rays contribute significantly to the diagnosis and treatment of patients and utilise teleradiology for specialist consultation should consider adopting digital CR imaging. 2. Radiographer training. All sites where X-ray services are utilised, whether from MSF or the MOH, should arrange for an initial visit by a radiographer for
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
88
Journal of Telemedicine and Telecare 20(2)
evaluation and training, followed by regular visits (at least yearly) for continuation of training. An evaluation of the X-ray department, radiographic technique and all images, not only those sent for teleradiology, should be used to identify the factors limiting image quality, so that improvements can be made. Radiographer training should include: (1) radiographic technique: exposure selection, patient positioning; (2) processing technique, CR post-processing; (3) implementation of a routine QA programme; (4) film evaluation and image critique: how to identify a high quality diagnostic image (anatomy coverage, desirable contrast/density) and how to evaluate each image against this standard. 3. Use of X-ray departments where quality has proved to be poor. If the majority of images from MOH X-ray services have proved to be of consistently poor quality, then MSF should consider whether the further utilisation of these services is justified. That is, without an investment to improve quality, potential misdiagnoses may be detrimental to the care of patients. Even though we have not evaluated the outcome of poor quality X-rays on management, it is well known that poor X-ray quality has a significant impact on the ability of radiologists and treating clinicians to interpret the image and manage patients. This is particularly true for teleradiology as the radiologists are not available on site to request repeat imaging or enforce quality standards. QA is critical for identifying weaknesses in diagnostic practices. QA of X-rays is feasible even for teleradiology and can highlight areas where important interventions may yield significant improvements. MSF sites performing poorly for quality must consider an upgrade in equipment and radiographer training as a priority, or should consider discontinuing referrals to these imaging departments. However desirable, the shift towards wider implementation of CR in resource-limited settings occurs slowly in practice. While there may be several reasons for this, the momentum behind this shift must be maintained. This will depend on continued publication of data on current practice from countries with limited resources.
2. Wootton R, Wu WI, Bonnardot L. Nucleating the development of telemedicine to support healthcare workers in resource-limited settings: a new approach. J Telemed Telecare 2013;19:411–417. 3. Alt CD, Engelmann D, Schenk JP, Troeger J. Quality control of thoracic X-rays in children in diagnostic centers with and without pediatric-radiologic competence. Rofo 2006;178:191–9. [German]. 4. Suric´ Mihic´ M, Mestrovic´ T, Prlic´ I, Suric´ D. Importance of quality assurance program implementation in conventional diagnostic radiology. Coll Antropol 2008;32(Suppl. 2): 181–4. 5. International Atomic Energy Agency. Quality Assurance in Paediatric Radiological Procedures L09. See http:// www.docstoc.com/docs/120742798/PAEDIATRIC-L09quality-assurance-in-paediatric-radiological-procedures (last checked 27 November 2013). 6. Corr P, Couper I, Beningfield SJ, Mars M. A simple telemedicine system using a digital camera. J Telemed Telecare 2000;6:233–6. 7. Whitehouse RW. Use of digital cameras for radiographs: how to get the best pictures. J R Soc Med 1999;92:178–82. 8. Szot A, Jacobson FL, Munn S, et al. Diagnostic accuracy of chest X-rays acquired using a digital camera for low-cost teleradiology. Int J Med Inform 2004;73:65–73. 9. Kogon PL, Lumsden R. How do you critique your radiographs? J Can Chiropr Assoc 1993;37:230–232. 10. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74. 11. Moran B, Upton J, Cooney P, Malone JF. A practical approach to a quality assurance programme for radiography at the constancy check level. Radiat Prot Dosimetry 1995;57:263–267. 12. Gallet JM, Reed MH, Hlady J. A novel quality assurance method in a university teaching paediatric radiology department. Br J Radiol 2000;73:843–6. 13. Cook JV. Radiation protection and quality assurance in paediatric radiology. Imaging 2001;13:229–238. 14. Dunn MA, Rogers AT. X-ray film reject analysis as a quality indicator. Radiography 1998;4:29–31. 15. Carmichael JHE, Maccia C, Moores BM, et al. European Guidelines on Quality Criteria for Diagnostic Radiographic Images. Luxembourg: European Commission, DirectorateGeneral XII: Science, Research and Development, 1996. 16. Conference of Radiation Control Program Directors. QA Collectible. Repeat Analysis. Available from http:// www.crcpd.org/Pubs/QAs.aspx (last checked 15 December 2013).
References 1. Me´decins Sans Frontie`res. International Activity Report 2012 – overview of activities. See http://www.msf.org/ international-activity-report-2012-overview-activities (last checked 27 November 2013).
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
Quality assessment of X-rays interpreted via teleradiology for Me´decins Sans Frontie`res
Journal of Telemedicine and Telecare 2014, Vol. 20(2) 82–88 ! The Author(s) 2014 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/1357633X14524153 jtt.sagepub.com
Saskia Spijker1, Savvas Andronikou2,3,4, Cara Kosack1, Richard Wootton5,6, Maryline Bonnet7 and Nathalie Lemmens8
Summary Me´decins Sans Frontie`res (MSF) is a humanitarian organisation which provides emergency medical aid in challenging settings; field staff often diagnose and treat patients using limited resources and without the expertise of specialists. Teleradiology is available for MSF sites which use digital computed radiography (CR) imaging or conventional film and chemistry. We conducted a retrospective study of the quality of X-rays utilised by MSF for teleradiology diagnosis over a one-year period. All plain X-ray examinations referred for interpretation using two MSF teleradiology platforms in 2012 were assessed against 15 image criteria and further evaluated as being either diagnostic or non-diagnostic. The sites studied sent an average of 115 images (range 10452). Images were a mixture of chest, skeletal and abdominal radiographs. The majority of the images were CR (n ¼ 597, 74%). Three sites were MSF/Epicentre installed and operated (Epicentre is a research facility affiliated with MSF); five sites were operated by the ministry of health, imaging patients referred by MSF. The sites performing poorest for quality were all facilities which used film and chemistry (53% non-diagnostic images). The sites performing better for quality were facilities which used CR digital imaging (12% non-diagnostic images), two of which had also undergone radiographer training. Our study suggests that transitioning to CR digital imaging has the potential to improve image quality compared to film and chemistry. Radiography training should be made a priority for all sites with X-ray services. The continued utilisation of X-ray services by MSF where images have proven to be consistently poor should be re-considered. Accepted: 17 December 2013
Introduction Me´decins Sans Frontie`res (MSF) is a non-governmental humanitarian organisation that responds to emergency situations and provides medical assistance to those in need. MSF teams provide medical emergency aid in difficult settings and staff often have to diagnose and treat patients with inadequate resources.1 Human resource shortages are common and the expertise of specialists such as radiologists is often lacking. Radiology is available in many locations where MSF provides medical services and teleradiology is available at all sites that have an Internet connection. Teleradiology is available both for sites that use digital computed radiography (CR) imaging and for those using film and chemistry followed by digitisation with a digital camera. MSF utilises two platforms for teleradiology: MSF’s own general telemedicine system (based on the Collegium Telemedicus2) and vRad (Virtual Radiologic Corporation, Eden Prairie, MN, USA), a commercial teleradiology company offering pro-bono services to MSF. Some of the referring X-ray departments are MSF installed and operated, while others are ministry of health (MOH) facilities within a hospital or referral centre. In all cases the purpose of teleradiology is to
obtain assistance in reaching a diagnosis and deciding the best clinical management of the patient. Quality assurance (QA) in radiology is an important component in ensuring high image quality.3 Evaluating image quality can identify and address the causes of poor quality images.4 These may be due to limitations in the competence of the radiographers (X-ray technicians), equipment problems, or a combination of both.5 QA in teleradiology should always include the quality of the radiographic image. This is particularly important in
1
Me´decins Sans Frontie`res International, Amsterdam, The Netherlands Department of Radiology, Faculty of Health Sciences, University of the Witwatersrand, South Africa 3 World Federation of Paediatric Imaging, Reston, USA 4 South African Medical Unit, Me´decins Sans Frontie`res, South Africa 5 Norwegian Centre for Integrated Care and Telemedicine, University Hospital of North Norway, Tromsø, Norway 6 Faculty of Health Sciences, University of Tromsø, Norway 7 Epicentre, Geneva, Switzerland 8 Onze Lieve Vrouwe Gasthuis, Amsterdam, The Netherlands 2
Corresponding author: Saskia Spijker, Plantage Middenlaan 14, 1018 DD, Amsterdam, The Netherlands. Email: [email protected]
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
Spijker et al.
83
resource-limited settings where there are greater challenges to achieving high quality images. The aim of the present study was to assess the images sent for teleradiology from routine MSF field sites.
Methods We conducted a retrospective study to determine the quality of X-rays utilised by MSF for teleradiology diagnosis over a one-year period. All plain X-ray examinations referred for interpretation using MSF teleradiology platforms in 2012 were assessed. Sites which sent fewer than 10 X-rays for teleradiology consultation in 2012 were excluded from the analysis. At sites using CR, X-ray images were produced in the medical imaging file format DICOM (Digital Imaging and Communications in Medicine) and were transmitted for teleradiology in this format. All the sites with CR imaging had Carestream CR readers. At sites using film and chemistry, hard copy X-ray films were digitised to JPEG files using a digital camera before transmission (JPEG is a commonly used method of lossy compression for digital photography). Sites using this method used their own digital camera and were encouraged to follow a protocol developed by MSF, based on previous studies about digitising X-ray films.6–8 The protocol suggested an image size of 3.5–5 Mpixel, high compression setting and exposure compensation setting of þ1.3 EV. All X-ray images were transmitted anonymously with a clear patient history. Images sent to vRad were transmitted via a secure Virtual Private Network (VPN) and cases transmitted through the MSF Collegium Telemedicus system were sent via Secure Messaging.
Quality assessment The initial quality assessment was performed by Observer 1 (a radiographer with more than 15 years experience). All images were viewed on a flat-screen monitor (MultiSync LCD 3090WQXi, NEC) at approximately 350 cd/m2 brightness. Each image was evaluated against 15 criteria. The criteria fell into six categories: positioning (1. rotation, 2. angulation, 3. other); collimation (4. limited anatomy, 5. excessive anatomy); exposure (6. contrast too high, 7. contrast too low, 8. density too high, 9. density too low); artefacts (10. motion, 11. other); markers (12. incorrect, 13. misplaced, 14. absent) or other (15. others). An image could be marked as being unacceptable in more than one criterion in a category. For example, a chest X-ray could be marked twice in the exposure group if it was evaluated as being too low in density and also too low in contrast, or twice for positioning if it was both rotated and with poor inspiration. In addition to being considered against these criteria, each image was evaluated as being of diagnostic or nondiagnostic quality overall. In view of the subjective nature of assessing quality in X-ray images, a quality reference guide was developed and used to determine the point at which an image would become non-diagnostic.
For example, a lateral wrist X-ray could be slightly rotated, have excessive collimation and be missing a marker, but could still be considered as a diagnostic image overall. On the other hand, a PA chest X-ray that was too high in density (too dark) but otherwise technically perfect would be considered non-diagnostic because this one factor limited its ability to be interpreted. Under normal circumstances, all X-ray images should be clearly marked with a unique patient identification, either name or file number, and with the patient’s date of birth, date of the X-ray and clinic/hospital identifier.9 This is a minimum legal requirement, but it was not evaluated in the present study because all MSF teleradiology images are sent without patient identifiers in accordance with the organisation’s privacy policy.
Data analysis We calculated the overall proportion and proportion per site of non-diagnostic film images and non-diagnostic CR digital images with corresponding 95% confidence intervals (CIs). These combined proportions were compared using the chi-squared test. The proportion of film and CR digital images requiring improvement was calculated for each group of quality assessment criteria. A random sample of the X-ray images was read independently by a second observer (a radiographer also with more than 15 years experience) using the same monitor as observer 1, in order to determine the agreement in the overall quality assessment between the two observers. The sample size was determined by assuming an expected kappa of 0.7, with a proportion of 25% of non-diagnostic X-rays by observer 1, 20% of discordance between the two observers, 10% precision and a 95% CI. This gave a minimum sample size of 245 images. Weighted kappa values were calculated with their 95% CIs and were interpreted as follows: kappa < 0 was considered to indicate no agreement; kappa ¼ 0.0 to 0.20 as slight agreement; kappa ¼ 0.21 to 0.40 as fair agreement; kappa ¼ 0.41 to 0.60 as moderate agreement; kappa ¼ 0.61 to 0.80 as substantial agreement; kappa ¼ 0.81 to 1.00 as almost perfect agreement.10 Data were analysed using a standard package (Stata 12.1, College Station, Texas, USA).
Results During the one-year study period, a total of 818 plain Xray images were sent for teleradiology consultation from 14 sites. Of the 14 sites, seven sites sent fewer than 10 images each and these sites were excluded from further analysis. Thus, the final sample concerned 806 images. The seven included sites sent an average of 115 images (range 10-452). Images were a mixture of chest, skeletal and abdominal radiographs, see Table 1. The majority of the images were CR: 74% (n ¼ 597) were CR and 26% (n ¼ 209) were film. The 806 images were from 623 individual patients; 64% (n ¼ 397) were adults and 36% (n ¼ 226) were children aged 15 years or under.
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
84
Journal of Telemedicine and Telecare 20(2)
Table 1. Sites sending at least 10 X-ray cases for teleradiology in 2012. Note that one site (Central Asia 2) sent both film and CR images – these are listed separately. No. of images
No. of non-diagnostic images
Percentage of non-diagnostic images
95% confidence interval
MOH MOH MOH MOH MOH
23 44 89 43 10 209
20 25 47 17 2 111
87 57 53 40 20 53
68–96 41–70 42–63 25–54 3–51 46–59
Epicentrea MSFb MSF
128 17 452 597
19 2 48 69
15 12 11 12
9–21 2–33 8–13 9–14
Facility Film Central Asia 1 Sub-Saharan Africa Central Asia 2 Sub-Saharan Africa Southeast Asia 1 Total CR Sub-Saharan Africa Central Asia 2 Sub-Saharan Africa Total
1 2
3 4
a
Research facility affiliated with MSF, utilising the MOH radiology department. Facility installed and staff trained by MSF, subsequently handed over to MOH.
b
Figure 1. Proportion of non-diagnostic images at sites using film and CR.
Primary outcome measure
Individual image criteria
The sites using film and chemistry for X-ray imaging had significantly more non-diagnostic images (53%, 95% CI 46-60) than sites with CR imaging (12%, 95% CI 9–14), see Table 1. This difference was significant (P < 0.001). Sites using film and chemistry tended to submit fewer cases for teleradiology, see Figure 1. Sites using CR tended to be those operated directly by MSF, see Figure 2. There was good agreement between the two observers in their assessments of film images (kappa ¼ 0.67, 95% CI 0.51–0.83) and almost perfect agreement for CR images (kappa ¼ 0.91, 95% CI 0.79–1.00).
Of the 209 film images, a total of 366 criteria were marked as requiring improvement. Of these images, the criteria marked most frequently were in the exposure category (68%, n ¼ 143), see Table 2. Of the 597 CR images, a total of 311 criteria were marked as requiring improvement. The criteria marked most frequently were in the collimation category (19%, n ¼ 116). The results of the individual criteria evaluated for improvement are shown in Figure 3. Collimation was considered an area for improvement in 25% (n ¼ 53) of the film images with limited
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
Spijker et al.
85
Figure 2. Proportion of non-diagnostic images at MOH and MSF sites. The site marked with an asterisk was originally installed and the staff trained by MSF, but was subsequently handed over to the MOH.
anatomy coverage being more common than excessive anatomy coverage (23% and 2% respectively). CR images were slightly better for collimation (19%, n ¼ 116) demonstrating almost equal limited and excessive anatomy coverage (9% and 10% respectively). In all instances of film and CR images with limited anatomy coverage, the radiographers had not recognised that the anatomy on the acquired X-ray was incomplete and had failed to repeat the film to ensure that all anatomy was covered. Inadequate exposure was the reason most frequently evaluated on film images as requiring improvement with too high contrast and too high density being the most common individual reasons. Both film and CR images are equally vulnerable to motion artefact and this was demonstrated on 6% (n ¼ 12) of the film images and 4% (n ¼ 24) of the CR images. Poor processing technique was included in the ‘artefacts-other’ category and this was the greatest contributor to this category. 24% (n ¼ 51) of film images were assessed as demonstrating artefacts in the ‘artefacts-other’ category compared with 1% (n ¼ 7) of CR images. In CR, processing artefacts rarely occur because there is no film and chemistry component. Absent, misplaced or incorrect markers occurred in 42% (n ¼ 89) of film images. CR images were marked against the marker criteria group on 6% of the images (n ¼ 40); 26 of these images were missing a side marker and the opportunity to add the marker at a postprocessing step was missed. The absence of a side marker would not necessarily cause a radiograph to be evaluated as non-diagnostic. However, use of side markers represents meticulous technique that is critical for both patient care and for medico-legal protection.
Discussion Quality assurance in radiography involves QA of the diagnostic X-ray equipment and review of the quality of the radiographs produced.11,12 Even though no single individual group or scientific/professional society contains all of the skills or knowledge required to perform QA efficiently11 it is important that all hospitals or clinics using radiography perform some kind of QA within their available resources. Good radiographic technique is the main factor in improving image quality.13 Experienced and dedicated staff are a key factor, but in the context of humanitarian aid agencies working in resource-limited settings, this can be a challenge. Wherever X-rays are taken, QA of radiographic quality is feasible and can be performed either by the people taking the radiographs (the radiographer) or by those interpreting them (the radiologist). One of the primary responsibilities of the radiographer is to ensure an adequate level of image quality before submitting the image to the radiologist for interpretation. In radiology departments, QA is generally performed by radiographers or radiologists undertaking a film reject analysis. The reject film rate is a good quality indicator.14 While there is no universally-agreed cut-off for an acceptable reject film rate,15 some advise that the overall rate should remain below 10%.16 Caution should be given, however, to using the reject rate as the sole representation of the quality of a radiology department because it is sensitive to examination type.14 The role of such an analysis should be to isolate specific quality problems that result in images that need to be rejected or repeated.14 Rejection rate is difficult to determine in teleradiology where the rejected hard copy films or CR images cannot be accessed, i.e. other images may have been taken and then discarded as
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
Journal of Telemedicine and Telecare 20(2) inadequate before the teleradiology transmission. On the other hand, the lack of experienced or supervised radiography staff may result in inadequate images being accepted and sent for teleradiology interpretation without any repetition. Although our quality assessment of images sent for teleradiology was not a film reject analysis, it employed the same principles.
87 42
26 4
1 –
10 2
4 2 2 0 – – Absent Misplaced
7 1 26 4 6 1 30 5
3 1
6 1
4 2
8 1
Film Percentage of 209 images Total group % CR Percentage of 597 images Total group %
7 3 7 48 8 10
Other Angulation Rotation
*each image could be marked against none or several criteria or twice within one category.
60 10
5 2
57 27 68 0 – 10
18 9
51 24
17 8
12 6 30 24 4 5
51 24
1 – 42 4 – 6
Film vs. CR
48 23 25 56 9 19
Incorrect Motion Contrast too high Limited anatomy
Excessive anatomy
Exposure Collimation Positioning
Table 2. Criteria for improvement in 806 plain X-ray images (film ¼ 209; CR ¼ 597).*
Contrast too low
Density too high
Density too low
Artefacts
Other
Markers
Other
86
The five sites performing the worst for quality (>20% nondiagnostic) were all MOH facilities using film and chemistry. The most common reasons for poor quality were inadequate exposure and artefacts due to processing. These areas all directly relate to the difficulty in obtaining good image quality using film and chemistry, where far greater technical knowledge and experience is required of radiographers in exposure selection and processing technique. Successful film processing also relies on a clean water supply and stable electricity to regulate the water temperature in the water bath surrounding the chemistry. Both of these factors are challenging in resource-limited settings. The three sites performing better for quality (15% or less non-diagnostic) were using CR digital imaging and were MSF/Epicentre sites. While the image quality was significantly higher at those sites, the percentage of non-diagnostic images still leaves room for improvement. There were far fewer problems with exposure due to the greater latitude within a digital system to compensate for exposure errors and the complete avoidance of contrast/density and artefact faults encountered with chemical processing of film. The images that were evaluated as non-diagnostic by the sites with CR were predominantly due to collimation (both limited and excessive demonstration of anatomy), positioning (particularly rotation of anatomy) and post-processing (not setting best contrast/density digitally). These factors all reflect the technique of the radiographers and support the need for continued training of radiographers in how to identify a high quality X-ray image. The results of the study show that the departments which sent more images for teleradiology also produced better quality images. However, we do not know what proportion of each department’s workload was sent for teleradiology.
Radiographer training The three sites with CR that performed best for quality were sites where teleradiology services were planned as a part of the X-ray installation. Two of these sites (SubSaharan Africa 4 and Central Asia 2 with CR) had also received training by an MSF expatriate radiographer (at the time of the X-ray machine and CR installation) and this probably contributed to both sites reporting the lowest percentages of non-diagnostic images. Staff at the site Sub-Saharan Africa 3 had also received technical training on the CR digital reader and teleradiology transmission, but this training did not extend to radiographic
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
Spijker et al.
87
Figure 3. Criteria for improvement. Each image could be marked against none or several criteria (see text).
technique inside the X-ray room itself. CR digital imaging is likely to contribute to a higher level of image quality but with extended radiographer training this could be improved even further. Where MSF utilise MOH X-ray facilities, this is usually done on a referral basis without the possibility of influencing equipment choices or the possibility for conducting training. This probably contributed to the significant difference in image quality between MSF sites with digital imaging where training had taken place and MOH sites using film and chemistry where no training had taken place.
been followed and the digitised image evaluated for quality was probably worse than the original film. This is less of a problem for images considered non-diagnostic due to limited anatomy or excessive artefacts, which are not influenced by camera digitisation. Our study, however, was evaluating the end result, i.e. the image sent to the radiologist for interpretation on the teleradiology platform. Investigating the importance of good camera technique in the digitisation process is an area for potential future investigation if conversion to CR is not possible.
Conclusion Limitations The present study had certain limitations. For example, image quality assessment was performed by a single observer and might therefore have been biased. However, an independent double-reading of a sample of the images by a second observer demonstrated substantial agreement for film images and almost perfect agreement for CR images. Nonetheless, evaluating an image is a subjective process in the absence of a gold standard. The difference in the kappa values observed between film and CR images could be explained by the smaller sample of film images. It was not always clear when viewing digitised film images (JPEG format) if the quality reflected the true quality of the original X-ray film or the influence of the digitisation process itself. Lung markings visible on an X-ray film may be less visible once digitised if the exposure value of the camera was set incorrectly, the flash was used, or if extraneous light around the X-ray was included in the field of view. It is difficult to know which sites strictly adhered to the protocol provided on how to digitise an X-ray film. In some cases it was very obvious that the protocol had not
The multi-country data that MSF has access to through its routine operations demonstrated a clear difference in quality between film and digital CR images. Radiographer training that was given to the MSF sites with CR imaging probably contributed to this difference in image quality. The following recommendations for MSF and other users of X-ray in resource-limited settings can be made: 1. Transition to CR imaging. Our assessment suggests that adopting CR imaging has the potential to improve image quality compared to film and chemistry, due to the technology compensating somewhat for radiographic variables (exposure, film processing procedures). This is a major advantage in resourcelimited settings. Sites where X-rays contribute significantly to the diagnosis and treatment of patients and utilise teleradiology for specialist consultation should consider adopting digital CR imaging. 2. Radiographer training. All sites where X-ray services are utilised, whether from MSF or the MOH, should arrange for an initial visit by a radiographer for
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
88
Journal of Telemedicine and Telecare 20(2)
evaluation and training, followed by regular visits (at least yearly) for continuation of training. An evaluation of the X-ray department, radiographic technique and all images, not only those sent for teleradiology, should be used to identify the factors limiting image quality, so that improvements can be made. Radiographer training should include: (1) radiographic technique: exposure selection, patient positioning; (2) processing technique, CR post-processing; (3) implementation of a routine QA programme; (4) film evaluation and image critique: how to identify a high quality diagnostic image (anatomy coverage, desirable contrast/density) and how to evaluate each image against this standard. 3. Use of X-ray departments where quality has proved to be poor. If the majority of images from MOH X-ray services have proved to be of consistently poor quality, then MSF should consider whether the further utilisation of these services is justified. That is, without an investment to improve quality, potential misdiagnoses may be detrimental to the care of patients. Even though we have not evaluated the outcome of poor quality X-rays on management, it is well known that poor X-ray quality has a significant impact on the ability of radiologists and treating clinicians to interpret the image and manage patients. This is particularly true for teleradiology as the radiologists are not available on site to request repeat imaging or enforce quality standards. QA is critical for identifying weaknesses in diagnostic practices. QA of X-rays is feasible even for teleradiology and can highlight areas where important interventions may yield significant improvements. MSF sites performing poorly for quality must consider an upgrade in equipment and radiographer training as a priority, or should consider discontinuing referrals to these imaging departments. However desirable, the shift towards wider implementation of CR in resource-limited settings occurs slowly in practice. While there may be several reasons for this, the momentum behind this shift must be maintained. This will depend on continued publication of data on current practice from countries with limited resources.
2. Wootton R, Wu WI, Bonnardot L. Nucleating the development of telemedicine to support healthcare workers in resource-limited settings: a new approach. J Telemed Telecare 2013;19:411–417. 3. Alt CD, Engelmann D, Schenk JP, Troeger J. Quality control of thoracic X-rays in children in diagnostic centers with and without pediatric-radiologic competence. Rofo 2006;178:191–9. [German]. 4. Suric´ Mihic´ M, Mestrovic´ T, Prlic´ I, Suric´ D. Importance of quality assurance program implementation in conventional diagnostic radiology. Coll Antropol 2008;32(Suppl. 2): 181–4. 5. International Atomic Energy Agency. Quality Assurance in Paediatric Radiological Procedures L09. See http:// www.docstoc.com/docs/120742798/PAEDIATRIC-L09quality-assurance-in-paediatric-radiological-procedures (last checked 27 November 2013). 6. Corr P, Couper I, Beningfield SJ, Mars M. A simple telemedicine system using a digital camera. J Telemed Telecare 2000;6:233–6. 7. Whitehouse RW. Use of digital cameras for radiographs: how to get the best pictures. J R Soc Med 1999;92:178–82. 8. Szot A, Jacobson FL, Munn S, et al. Diagnostic accuracy of chest X-rays acquired using a digital camera for low-cost teleradiology. Int J Med Inform 2004;73:65–73. 9. Kogon PL, Lumsden R. How do you critique your radiographs? J Can Chiropr Assoc 1993;37:230–232. 10. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74. 11. Moran B, Upton J, Cooney P, Malone JF. A practical approach to a quality assurance programme for radiography at the constancy check level. Radiat Prot Dosimetry 1995;57:263–267. 12. Gallet JM, Reed MH, Hlady J. A novel quality assurance method in a university teaching paediatric radiology department. Br J Radiol 2000;73:843–6. 13. Cook JV. Radiation protection and quality assurance in paediatric radiology. Imaging 2001;13:229–238. 14. Dunn MA, Rogers AT. X-ray film reject analysis as a quality indicator. Radiography 1998;4:29–31. 15. Carmichael JHE, Maccia C, Moores BM, et al. European Guidelines on Quality Criteria for Diagnostic Radiographic Images. Luxembourg: European Commission, DirectorateGeneral XII: Science, Research and Development, 1996. 16. Conference of Radiation Control Program Directors. QA Collectible. Repeat Analysis. Available from http:// www.crcpd.org/Pubs/QAs.aspx (last checked 15 December 2013).
References 1. Me´decins Sans Frontie`res. International Activity Report 2012 – overview of activities. See http://www.msf.org/ international-activity-report-2012-overview-activities (last checked 27 November 2013).
Downloaded from jtt.sagepub.com at UNIV OF WESTERN ONTARIO on March 8, 2015
