Journal of the ICRU Vol 6 No 2 (2006) Report 76 Oxford University Press
doi:10.1093/jicru/ndl029
4
MEASUREMENT ANALYSIS METHODS
4.1
CONTROL CHARTS
The quality of a routine measurement system can be monitored, and where necessary corrected or improved, by the statistical analysis of measurement results over long periods of time (van Dijk, 1998). The mean values, variances, and shapes of distributions for monthly, quarterly, and/or annual samples of results for the total data set, or for subsets, can be anticipated to be less variable than the mean and variance for the total data set. The analysis might also be carried out for different dose levels or activity concentrations. In addition to the use of statistical analysis for examining the dosimetric data for quality-assurance purposes, there can also be a need to demonstrate to authorities that the dosimetry provider has maintained the required level of consistency with national standards over an extended time period. The calibration records will also indicate any deviations from the control limits and notations as to how the deviations were dealt with (see Sections 4.4 and 4.5). 4.3
TREATMENT OF OUTLIERS
During the performance of a series of measurements, something unexpected or unknown might happen that causes incorrect measurements. If there is no obvious indication of a problem at the time of the measurement, there might be no reason for rejecting the result of the measurement at that point. However, when the data are plotted in the form of a control chart or when a table of data is examined, it might be observed that one or more points differ significantly from the others. Individual data points that are clearly different from the expected distribution for the measurand are called outliers. If the unexpected event causing the incorrect measurement has been observed, e.g., a transient in the line voltage or a significant change in room temperature, then the measurement affected by this event can be classified as an outlier. Outliers of this type can be considered for removal from a data set before identifying the cause, because inclusion might
Ó International Commission on Radiation Units and Measurements 2006
Downloaded from http://jicru.oxfordjournals.org/ at University of Guelph on October 4, 2012
Graphical presentation of data can provide useful information about variations and trends. This type of presentation is also well suited for demonstrations to laboratory personnel and others who might need to examine the stability of calibration measurements as a function of time. In addition, a large amount of data can be presented graphically in a control chart format. Comparisons between measurements performed many years ago can easily be compared with recent data. Although most of these same characteristics can be attributed to the format of data presented in the tabular form, it is often easier to detect small problems with the data when presented in a graphical form. In a control chart, data can be plotted either as ratios relative to a reference value or simply as the numerical values of the measured quantity. Normally, these values are plotted as a function of time. Graphical indicators of deviation levels that should trigger investigation or action can be plotted above and below an expected value for the measurand. A typical control chart is shown in Figure 4.1. As an example, two indicators can be placed at positions that represent expected statistical uncertainty for the measurement. For instance, indicators can be placed at set percentage variations from the reference value. A data point that exceeds this indicator is easily visible and can trigger an investigation of a possible change in response. A second set of indicators can be placed at a larger variation value, and data points falling outside these indicators might require a work-stoppage and correctiveaction plan. It is appropriate to record notes as to the cause of the deviation and the rectification of the problem that caused that deviation. In some situations, the indicators or control levels might not be symmetric with respect to the mean or reference value. This could occur when it is important not to allow the measurement to fall below a certain value. For example, the dose needed to sterilize a medical product is a critical parameter, and serious consequences result if the dose is not delivered.
4.2 LONG-TERM STATISTICAL ANALYSIS OF DATA
MEASUREMENT QUALITY ASSURANCE FOR IONIZING RADIATION DOSIMETRY
For example, in a report the correct value should be entered alongside the value that has been determined to be incorrect. All such alterations to records should be signed and dated by the person making the correction. Similar techniques can be applied to electronic records. Many software programs contain features that allow the strikeout of a word or a number to remain visible and to be stored along with the modified data. Records of errors or, in general, of malfunctions in any part of the process of measurement should be retained and periodically checked, in order to avoid recurrences of the same error. These records can be used later for investigations or in-house reviews. The laboratory must have procedures in place for dealing with mistakes. These procedures should ensure that responsibilities of staff members are defined and that action is taken when a problem is uncovered. Staff members should have the authority to stop work when a significant problem comes to their attention. Once a mistake has been uncovered, either within the facility or by a client or customer of the measurement facility, corrective actions should be taken, retrospectively, as necessary. When appropriate, the clients or customers shall be notified. Appropriate verifications of measurement quality should be performed before work is authorized to continue. Several methods have been found to be useful in reducing the occurrence of errors. Informal audits of measurement quality performed by a member of
distort calculations of the mean and standard deviation. If no causative event can be associated with an apparent outlier and no deviation from standard test conditions of temperature, pressure, etc. is evident, then additional data should be taken. Statistical methods should be used to test whether the result of measurement can be considered an outlier; see for example, Huber (1981) and Rousseeuw and Leroy (1987).
4.4 ERROR (QUALITY ANOMALY) CORRECTION PROCEDURES One of the most important activities of a facility for the calibration and measurement of ionizing radiation is the awareness and minimization of errors. It cannot be assumed that mistakes will never happen. In fact, it is a more realistic viewpoint to assume that errors will occur, but that every effort will be taken to minimize their frequency of occurrence and their potential for negative effects. A consequence of this assumption is the need to develop techniques for dealing with mistakes, correcting erroneous results, and revising procedures so that the problem does not recur. An advisable method to be followed in case of mistakes that have been discovered in written documentation is that the record of the mistake should be crossed out, not erased or made illegible. 28
Downloaded from http://jicru.oxfordjournals.org/ at University of Guelph on October 4, 2012
Figure 4.1. Thermoluminescent dosimeter (TLD) readings recorded as a function of time. The four plots indicate readings from the four TLD chips used in a particular design of personal dosimeter. Readings exceeding control limits may bear investigation. (Plot kindly supplied by B.A. Rathbone and S.E. Huneycutt of the Pacific Northwest National Laboratory).
MEASURMENT ANALYSIS METHODS
4.5
REVIEW AND APPROVAL OF RECORDS
It is essential that measurement records be subject to rigorous quality-assurance procedures and, when required, are compatible with national regulations and national data record-keeping requirements. The standard quality-assurance procedures of checking records for completeness, conformity, document control, duration of storage, safety, and back-up should be applied. Procedures must be clearly defined in order to be checked and approved by supervisory staff, and the procedure with the most current date is taken as the official version. A record is a document that states results that have been achieved or provides evidence of activities that have been performed. A record does not undergo the periodic review and updating activities used for procedures. However, it is subject to additional special requirements to ensure correctness. A procedure should be established to control technical records and quality records so that their identification, archiving, storage, and eventual disposal after specified retention times are assured. Quality records include reports from internal and external audits along with management reviews and documentation of corrective and preventive actions. Records are to be kept legible, secure, and held in confidence. Usually, records shall not be changed. If a record is subsequently found to have been inaccurate, a clearly identified corrected record is issued while retaining the original that has been marked as incorrect. The correction should not make the original data illegible or result in its deletion. It is advisable that an identification of the person responsible for the correction be placed alongside the correction. This also applies to records stored electronically. Configuration-control software can be used to make changes in a record without erasing the original entry, and electronic notes can be entered to identify the person who has made the correction.
4.6 QUALITY AUDIT AND QUALITY SYSTEM REVIEW Properly conducted internal and external audits and reviews are an essential part of any quality system. Their purpose is to check the effectiveness of the quality assurance system, which the procedures are clearly documented and are being followed, and to identify areas for improvement. As indicated in Figure 2.1, even in the early stages of developing the quality system an internal audit should be performed to ensure that the procedures developed for use in the laboratory are understood and are being followed by the staff. Quality audits shall be carried out regularly to ensure that the quality system as detailed in the documentation is fully implemented as described. Quality audits might need to be carried out following the investigation of an error. Internal audits should be carried out reasonably frequently or at least annually. External audits can be less frequent. Reviews are surveys of the quality system to determine if the system as described meets the quality-assurance policy requirements. Reviews would normally be carried out annually and give an opportunity to assess the overall effectiveness of the quality assurance system and to make modifications and revisions.
29
Downloaded from http://jicru.oxfordjournals.org/ at University of Guelph on October 4, 2012
Technical records include calibration reports, staff records and copies of issued reports, original observations, derived data, and sufficient information to establish an audit trail. The records retained should be sufficient to facilitate the identification of factors affecting the uncertainty and to enable the test to be repeated under conditions as close as possible to the original test. Therefore, all the data needed to reconstruct the calculations to achieve the result of the measurement should be systematically archived and kept for a predetermined time period in a secure location. The records should include the identification of persons responsible for sampling, performing each test or calibration, and checking results. Record keeping should fulfill the requirements for good practices of data management (ANSI/HPS, 1999). The main principles relating to data quality are that data are processed correctly in an unbiased manner, are collected for specific purposes, are adequate to fulfill these purposes, and are kept up to date.
the laboratory are effective in identifying potential problems. One staff member can be designated with the responsibility for reviewing procedures and requesting measurements (as if that person were a client or customer). Periodic round-robin comparisons with similar calibration facilities also serve as useful checks on measurement quality, and these interactions also encourage communication of common problems and successful solutions.
doi:10.1093/jicru/ndl029
4
MEASUREMENT ANALYSIS METHODS
4.1
CONTROL CHARTS
The quality of a routine measurement system can be monitored, and where necessary corrected or improved, by the statistical analysis of measurement results over long periods of time (van Dijk, 1998). The mean values, variances, and shapes of distributions for monthly, quarterly, and/or annual samples of results for the total data set, or for subsets, can be anticipated to be less variable than the mean and variance for the total data set. The analysis might also be carried out for different dose levels or activity concentrations. In addition to the use of statistical analysis for examining the dosimetric data for quality-assurance purposes, there can also be a need to demonstrate to authorities that the dosimetry provider has maintained the required level of consistency with national standards over an extended time period. The calibration records will also indicate any deviations from the control limits and notations as to how the deviations were dealt with (see Sections 4.4 and 4.5). 4.3
TREATMENT OF OUTLIERS
During the performance of a series of measurements, something unexpected or unknown might happen that causes incorrect measurements. If there is no obvious indication of a problem at the time of the measurement, there might be no reason for rejecting the result of the measurement at that point. However, when the data are plotted in the form of a control chart or when a table of data is examined, it might be observed that one or more points differ significantly from the others. Individual data points that are clearly different from the expected distribution for the measurand are called outliers. If the unexpected event causing the incorrect measurement has been observed, e.g., a transient in the line voltage or a significant change in room temperature, then the measurement affected by this event can be classified as an outlier. Outliers of this type can be considered for removal from a data set before identifying the cause, because inclusion might
Ó International Commission on Radiation Units and Measurements 2006
Downloaded from http://jicru.oxfordjournals.org/ at University of Guelph on October 4, 2012
Graphical presentation of data can provide useful information about variations and trends. This type of presentation is also well suited for demonstrations to laboratory personnel and others who might need to examine the stability of calibration measurements as a function of time. In addition, a large amount of data can be presented graphically in a control chart format. Comparisons between measurements performed many years ago can easily be compared with recent data. Although most of these same characteristics can be attributed to the format of data presented in the tabular form, it is often easier to detect small problems with the data when presented in a graphical form. In a control chart, data can be plotted either as ratios relative to a reference value or simply as the numerical values of the measured quantity. Normally, these values are plotted as a function of time. Graphical indicators of deviation levels that should trigger investigation or action can be plotted above and below an expected value for the measurand. A typical control chart is shown in Figure 4.1. As an example, two indicators can be placed at positions that represent expected statistical uncertainty for the measurement. For instance, indicators can be placed at set percentage variations from the reference value. A data point that exceeds this indicator is easily visible and can trigger an investigation of a possible change in response. A second set of indicators can be placed at a larger variation value, and data points falling outside these indicators might require a work-stoppage and correctiveaction plan. It is appropriate to record notes as to the cause of the deviation and the rectification of the problem that caused that deviation. In some situations, the indicators or control levels might not be symmetric with respect to the mean or reference value. This could occur when it is important not to allow the measurement to fall below a certain value. For example, the dose needed to sterilize a medical product is a critical parameter, and serious consequences result if the dose is not delivered.
4.2 LONG-TERM STATISTICAL ANALYSIS OF DATA
MEASUREMENT QUALITY ASSURANCE FOR IONIZING RADIATION DOSIMETRY
For example, in a report the correct value should be entered alongside the value that has been determined to be incorrect. All such alterations to records should be signed and dated by the person making the correction. Similar techniques can be applied to electronic records. Many software programs contain features that allow the strikeout of a word or a number to remain visible and to be stored along with the modified data. Records of errors or, in general, of malfunctions in any part of the process of measurement should be retained and periodically checked, in order to avoid recurrences of the same error. These records can be used later for investigations or in-house reviews. The laboratory must have procedures in place for dealing with mistakes. These procedures should ensure that responsibilities of staff members are defined and that action is taken when a problem is uncovered. Staff members should have the authority to stop work when a significant problem comes to their attention. Once a mistake has been uncovered, either within the facility or by a client or customer of the measurement facility, corrective actions should be taken, retrospectively, as necessary. When appropriate, the clients or customers shall be notified. Appropriate verifications of measurement quality should be performed before work is authorized to continue. Several methods have been found to be useful in reducing the occurrence of errors. Informal audits of measurement quality performed by a member of
distort calculations of the mean and standard deviation. If no causative event can be associated with an apparent outlier and no deviation from standard test conditions of temperature, pressure, etc. is evident, then additional data should be taken. Statistical methods should be used to test whether the result of measurement can be considered an outlier; see for example, Huber (1981) and Rousseeuw and Leroy (1987).
4.4 ERROR (QUALITY ANOMALY) CORRECTION PROCEDURES One of the most important activities of a facility for the calibration and measurement of ionizing radiation is the awareness and minimization of errors. It cannot be assumed that mistakes will never happen. In fact, it is a more realistic viewpoint to assume that errors will occur, but that every effort will be taken to minimize their frequency of occurrence and their potential for negative effects. A consequence of this assumption is the need to develop techniques for dealing with mistakes, correcting erroneous results, and revising procedures so that the problem does not recur. An advisable method to be followed in case of mistakes that have been discovered in written documentation is that the record of the mistake should be crossed out, not erased or made illegible. 28
Downloaded from http://jicru.oxfordjournals.org/ at University of Guelph on October 4, 2012
Figure 4.1. Thermoluminescent dosimeter (TLD) readings recorded as a function of time. The four plots indicate readings from the four TLD chips used in a particular design of personal dosimeter. Readings exceeding control limits may bear investigation. (Plot kindly supplied by B.A. Rathbone and S.E. Huneycutt of the Pacific Northwest National Laboratory).
MEASURMENT ANALYSIS METHODS
4.5
REVIEW AND APPROVAL OF RECORDS
It is essential that measurement records be subject to rigorous quality-assurance procedures and, when required, are compatible with national regulations and national data record-keeping requirements. The standard quality-assurance procedures of checking records for completeness, conformity, document control, duration of storage, safety, and back-up should be applied. Procedures must be clearly defined in order to be checked and approved by supervisory staff, and the procedure with the most current date is taken as the official version. A record is a document that states results that have been achieved or provides evidence of activities that have been performed. A record does not undergo the periodic review and updating activities used for procedures. However, it is subject to additional special requirements to ensure correctness. A procedure should be established to control technical records and quality records so that their identification, archiving, storage, and eventual disposal after specified retention times are assured. Quality records include reports from internal and external audits along with management reviews and documentation of corrective and preventive actions. Records are to be kept legible, secure, and held in confidence. Usually, records shall not be changed. If a record is subsequently found to have been inaccurate, a clearly identified corrected record is issued while retaining the original that has been marked as incorrect. The correction should not make the original data illegible or result in its deletion. It is advisable that an identification of the person responsible for the correction be placed alongside the correction. This also applies to records stored electronically. Configuration-control software can be used to make changes in a record without erasing the original entry, and electronic notes can be entered to identify the person who has made the correction.
4.6 QUALITY AUDIT AND QUALITY SYSTEM REVIEW Properly conducted internal and external audits and reviews are an essential part of any quality system. Their purpose is to check the effectiveness of the quality assurance system, which the procedures are clearly documented and are being followed, and to identify areas for improvement. As indicated in Figure 2.1, even in the early stages of developing the quality system an internal audit should be performed to ensure that the procedures developed for use in the laboratory are understood and are being followed by the staff. Quality audits shall be carried out regularly to ensure that the quality system as detailed in the documentation is fully implemented as described. Quality audits might need to be carried out following the investigation of an error. Internal audits should be carried out reasonably frequently or at least annually. External audits can be less frequent. Reviews are surveys of the quality system to determine if the system as described meets the quality-assurance policy requirements. Reviews would normally be carried out annually and give an opportunity to assess the overall effectiveness of the quality assurance system and to make modifications and revisions.
29
Downloaded from http://jicru.oxfordjournals.org/ at University of Guelph on October 4, 2012
Technical records include calibration reports, staff records and copies of issued reports, original observations, derived data, and sufficient information to establish an audit trail. The records retained should be sufficient to facilitate the identification of factors affecting the uncertainty and to enable the test to be repeated under conditions as close as possible to the original test. Therefore, all the data needed to reconstruct the calculations to achieve the result of the measurement should be systematically archived and kept for a predetermined time period in a secure location. The records should include the identification of persons responsible for sampling, performing each test or calibration, and checking results. Record keeping should fulfill the requirements for good practices of data management (ANSI/HPS, 1999). The main principles relating to data quality are that data are processed correctly in an unbiased manner, are collected for specific purposes, are adequate to fulfill these purposes, and are kept up to date.
the laboratory are effective in identifying potential problems. One staff member can be designated with the responsibility for reviewing procedures and requesting measurements (as if that person were a client or customer). Periodic round-robin comparisons with similar calibration facilities also serve as useful checks on measurement quality, and these interactions also encourage communication of common problems and successful solutions.
