533846
MSJ0010.1177/1352458514533846Multiple Sclerosis JournalJ Dong, Y Zhaoresearch-article2015
MULTIPLE SCLEROSIS MSJ JOURNAL
Short Report
Further investigation of safety monitoring guidelines based on magnetic resonance imaging lesion activity in multiple sclerosis clinical trials
Multiple Sclerosis Journal 2015, Vol. 21(1) 101–104 DOI: 10.1177/ 1352458514533846 © The Author(s), 2015. Reprints and permissions: http://www.sagepub.co.uk/ journalsPermissions.nav
Jing Dong, Yinshan Zhao, A John Petkau, David KB Li, Andrew Riddehough and Anthony Traboulsee
Abstract: We assess two modified guidelines for monitoring patient safety in multiple sclerosis (MS) trials. These guidelines flag patients with an increase in contrast enhancing lesion (CEL) count above a threshold over the CEL level 1–2 months earlier. We compare the new guidelines to the original guideline where the threshold is set according to the baseline by applying the guidelines to two previous studies. The odds ratios of a subsequent clinical relapse associated with meeting the CEL threshold based on the modified guidelines are similar to those based on the original guideline. There is a need for patient and cohort specific monitoring procedures. Keywords: Clinical trials methodology, multiple sclerosis, patient safety, magnetic resonance imaging, contrast-enhancing lesions Date received: 9 January 2014; revised: 1 April 2014; accepted: 6 April 2014 Introduction A safety guideline commonly used by Data Safety Monitoring Boards of multiple sclerosis (MS) clinical trials is to flag patients who, on a follow-up scan, have an increase of five or more contrast enhancing lesions (CELs) above the patient’s baseline level (“Rule of Five”).1,2 Elevated CEL counts meeting this guideline are associated with an increased risk of relapse in the subsequent 28 days.1,2 However, the sensitivity (i.e. the rate of meeting the threshold among those who had a subsequent relapse) can be low (under 0.30 for all thresholds considered in the previous study).1 As the follow-up time point becomes further removed from the baseline, an increase relative to the patient’s baseline level might also become less relevant. We assess the performance of two modified guidelines based on CEL count increases relative to recent scans. Materials and methods We refer to the original guideline as G1 where the threshold is set relative to the baseline level. The two modified guidelines are:
•• G2: the threshold is set relative to the most recent prior scan. •• G3: the threshold is set relative to the average CEL counts on the two prior scans. Using a threshold of five as an example, if the CEL count of a patient at the third visit is ≥5 above the CEL count on the patient’s second visit, this patient meets the guideline G2 at the third visit. A patient meets the guideline G3 at the third visit if the CEL count at this visit is ≥5 above the average CEL count on the patient’s first and second visits. In lieu of a missing CEL count at a previous visit, we use the last available count. We investigated threshold values from 1–15. We applied these guidelines to the same Phase II trials assessed previously.1 Cohort 1 consists of 167 relapsing patients.3 Cohort 2 consists of 185 secondary progressive (SP) patients.4 Patients from both studies had magnetic resonance imaging (MRI) visits at screening, baseline and every four weeks for 24 weeks. The clinical visits for relapse assessment roughly coincided with the MRI visits and there was at least one
Correspondence to: Yinshan Zhao MS/MRI Research Group, University of British Columbia, S195, UBC Hospital, 2211 Wesbrook Mall, Vancouver, BC, V6T 2B5, Canada. [email protected]. ubc.ca Jing Dong University of British Columbia, Canada Yinshan Zhao MS/MRI Research Group, University of British Columbia, Canada A John Petkau David KB Li Andrew Riddehough Anthony Traboulsee University of British Columbia, Canada Jing Dong and Yinshan Zhao contributed equally to this manuscript.
http://msj.sagepub.com 101
Downloaded from msj.sagepub.com at NORTHERN ARIZONA UNIVERSITY on June 10, 2015
Downloaded from msj.sagepub.com at NORTHERN ARIZONA UNIVERSITY on June 10, 2015
3
2
1
3
2
1
3
2
1
3
17/833 (2.0%) 6/833 (0.7%) 7/833 (0.8%)
31/833 (3.7%) 19/833 (2.3%) 17/833 (2.0%)
38/833 (4.6%) 25/833 (3.0%) 20/833 (2.4%)
64/833 (7.7%) 49/833 (5.9%) 52/833 (6.2%)
Scans met TH (%)
6/17 (35.3%) 4/6 (66.7%) 3/7 (42.9%)
9/31 (29.0%) 5/19 (26.3%) 5/17 (29.4%)
10/38 (26.3%) 6/25 (24.0%) 5/20 (25.0%)
13/64 (20.3%) 9/49 (18.4%) 10/52 (19.2%)
Met TH
91/816 (11.2%) 93/827 (11.2%) 94/826 (11.4%)
88/802 (11.0%) 92/814 (11.3%) 92/816 (11.3%)
87/795 (10.9%) 91/808 (11.3%) 92/813 (11.3%)
84/769 (10.9%) 88/784 (11.2%) 87/781 (11.1%)
Did not meet TH
Proportion with a relapse in 28 days
5.84
15.78
4.35
3.28
2.80
3.32
2.61
2.49
2.91
1.90
1.78
2.08
Crude ORa
4.44f (1.5–13.6) 16.44f (2.7–99.8) 5.64e (1.1–28.7)
3.25f (1.7–7.9) 2.51 (0.8–7.5) 2.89 (0.9–9.0)
2.83e (1.2–6.4) 2.25 (0.8–6.1) 2.25 (0.8–6.7)
1.97 (1.0–4.0) 1.63 (0.7–3.6) 1.73 (0.8–3.7)
Adjusted OR (CI)b
71/1032 (6.9%) 52/1032 (5.0%) 51/1032 (4.9%) 42/1032 (4.1%) 29/1032 (2.8%) 35/1032 (3.4%) 31/1032 (3.0%) 21/1032 (2.0%) 25/1032 (2.4%) 12/1032 (1.2%) 3/1032 (0.3%) 4/1032 (0.4%)
0.10 0.96 0.06 0.97 0.05 0.98 0.09 0.97 0.05 0.98 0.05 0.98 0.06 0.99 0.04 1.00 0.03 0.99
Scans met TH (%)
0.13 0.93 0.09 0.95 0.10 0.94
SENSc SPECd
2/12 (16.7%) 0/3 (0.0%) 1/4 (25.0%)
5/31 (16.1%) 2/21 (9.5%) 4/25 (16.0%)
5/42 (11.9%) 4/29 (13.8%) 5/35 (14.3%)
6/71 (8.5%) 7/52 (13.5%) 5/51 (9.8%)
Met TH
42/1020 (4.1%) 44/1029 (4.3%) 43/1028 (4.2%)
39/1001 (3.9%) 42/1011 (4.2%) 40/1007 (4.0%)
39/990 (3.9%) 40/1003 (4.0%) 39/997 (3.9%)
38/961 (4.0%) 37/980 (3.8%) 39/981 (4.0%)
Did not meet TH
Proportion with a relapse in 28 days
Secondary progressive cohort
7.64
0.00
4.66
4.60
2.43
4.74
4.09
3.85
3.30
2.63
3.96
2.24
Crude ORa
2.94 (0.4–21.3) 0.00 NAg 2.53 (0.2–37.2)
2.41 (0.7–8.6) 0.73 (0.1–4.3) 1.65 (0.4–7.0)
1.58 (0.5–5.4) 1.48 (0.4–5.7) 1.51 (0.4–5.5)
1.03 (0.3–3.3) 1.73 (0.6–5.1) 1 .02 (0.3–3.5)
Adjusted OR (CI)b
0.05 0.99 0.00 1.00 0.02 1.00
0.11 0.97 0.05 0.98 0.09 0.98
0.11 0.96 0.09 0.97 0.11 0.97
0.14 0.93 0.16 0.95 0.11 0.95
SENSc SPECd
CI: confidence interval; G: guidelines (G1: threshold relative to the baseline magnetic resonance imaging (MRI) scan, G2: threshold relative to the most recent prior MRI scan, G3: threshold relative to the average of the two prior MRIs); OR: odds ratio. aCrude ORs are based on the 2×2 table of observed frequencies. bAdjusted ORs are based on the final mixed-effects logistic model which included adjustment for: Relapsing (R) cohort: time of visit; secondary progressive (SP) cohort: baseline contrast enhancing lesion (CEL), number of pre-study relapses and SP duration. cSENS: sensitivity (true positive rate), number of true positives/(number of true positives + number of false negatives). dSPEC: specificity (true negative rate), number of true negatives/(number of true negatives + number of false positives). eORs with 0.05
MSJ0010.1177/1352458514533846Multiple Sclerosis JournalJ Dong, Y Zhaoresearch-article2015
MULTIPLE SCLEROSIS MSJ JOURNAL
Short Report
Further investigation of safety monitoring guidelines based on magnetic resonance imaging lesion activity in multiple sclerosis clinical trials
Multiple Sclerosis Journal 2015, Vol. 21(1) 101–104 DOI: 10.1177/ 1352458514533846 © The Author(s), 2015. Reprints and permissions: http://www.sagepub.co.uk/ journalsPermissions.nav
Jing Dong, Yinshan Zhao, A John Petkau, David KB Li, Andrew Riddehough and Anthony Traboulsee
Abstract: We assess two modified guidelines for monitoring patient safety in multiple sclerosis (MS) trials. These guidelines flag patients with an increase in contrast enhancing lesion (CEL) count above a threshold over the CEL level 1–2 months earlier. We compare the new guidelines to the original guideline where the threshold is set according to the baseline by applying the guidelines to two previous studies. The odds ratios of a subsequent clinical relapse associated with meeting the CEL threshold based on the modified guidelines are similar to those based on the original guideline. There is a need for patient and cohort specific monitoring procedures. Keywords: Clinical trials methodology, multiple sclerosis, patient safety, magnetic resonance imaging, contrast-enhancing lesions Date received: 9 January 2014; revised: 1 April 2014; accepted: 6 April 2014 Introduction A safety guideline commonly used by Data Safety Monitoring Boards of multiple sclerosis (MS) clinical trials is to flag patients who, on a follow-up scan, have an increase of five or more contrast enhancing lesions (CELs) above the patient’s baseline level (“Rule of Five”).1,2 Elevated CEL counts meeting this guideline are associated with an increased risk of relapse in the subsequent 28 days.1,2 However, the sensitivity (i.e. the rate of meeting the threshold among those who had a subsequent relapse) can be low (under 0.30 for all thresholds considered in the previous study).1 As the follow-up time point becomes further removed from the baseline, an increase relative to the patient’s baseline level might also become less relevant. We assess the performance of two modified guidelines based on CEL count increases relative to recent scans. Materials and methods We refer to the original guideline as G1 where the threshold is set relative to the baseline level. The two modified guidelines are:
•• G2: the threshold is set relative to the most recent prior scan. •• G3: the threshold is set relative to the average CEL counts on the two prior scans. Using a threshold of five as an example, if the CEL count of a patient at the third visit is ≥5 above the CEL count on the patient’s second visit, this patient meets the guideline G2 at the third visit. A patient meets the guideline G3 at the third visit if the CEL count at this visit is ≥5 above the average CEL count on the patient’s first and second visits. In lieu of a missing CEL count at a previous visit, we use the last available count. We investigated threshold values from 1–15. We applied these guidelines to the same Phase II trials assessed previously.1 Cohort 1 consists of 167 relapsing patients.3 Cohort 2 consists of 185 secondary progressive (SP) patients.4 Patients from both studies had magnetic resonance imaging (MRI) visits at screening, baseline and every four weeks for 24 weeks. The clinical visits for relapse assessment roughly coincided with the MRI visits and there was at least one
Correspondence to: Yinshan Zhao MS/MRI Research Group, University of British Columbia, S195, UBC Hospital, 2211 Wesbrook Mall, Vancouver, BC, V6T 2B5, Canada. [email protected]. ubc.ca Jing Dong University of British Columbia, Canada Yinshan Zhao MS/MRI Research Group, University of British Columbia, Canada A John Petkau David KB Li Andrew Riddehough Anthony Traboulsee University of British Columbia, Canada Jing Dong and Yinshan Zhao contributed equally to this manuscript.
http://msj.sagepub.com 101
Downloaded from msj.sagepub.com at NORTHERN ARIZONA UNIVERSITY on June 10, 2015
Downloaded from msj.sagepub.com at NORTHERN ARIZONA UNIVERSITY on June 10, 2015
3
2
1
3
2
1
3
2
1
3
17/833 (2.0%) 6/833 (0.7%) 7/833 (0.8%)
31/833 (3.7%) 19/833 (2.3%) 17/833 (2.0%)
38/833 (4.6%) 25/833 (3.0%) 20/833 (2.4%)
64/833 (7.7%) 49/833 (5.9%) 52/833 (6.2%)
Scans met TH (%)
6/17 (35.3%) 4/6 (66.7%) 3/7 (42.9%)
9/31 (29.0%) 5/19 (26.3%) 5/17 (29.4%)
10/38 (26.3%) 6/25 (24.0%) 5/20 (25.0%)
13/64 (20.3%) 9/49 (18.4%) 10/52 (19.2%)
Met TH
91/816 (11.2%) 93/827 (11.2%) 94/826 (11.4%)
88/802 (11.0%) 92/814 (11.3%) 92/816 (11.3%)
87/795 (10.9%) 91/808 (11.3%) 92/813 (11.3%)
84/769 (10.9%) 88/784 (11.2%) 87/781 (11.1%)
Did not meet TH
Proportion with a relapse in 28 days
5.84
15.78
4.35
3.28
2.80
3.32
2.61
2.49
2.91
1.90
1.78
2.08
Crude ORa
4.44f (1.5–13.6) 16.44f (2.7–99.8) 5.64e (1.1–28.7)
3.25f (1.7–7.9) 2.51 (0.8–7.5) 2.89 (0.9–9.0)
2.83e (1.2–6.4) 2.25 (0.8–6.1) 2.25 (0.8–6.7)
1.97 (1.0–4.0) 1.63 (0.7–3.6) 1.73 (0.8–3.7)
Adjusted OR (CI)b
71/1032 (6.9%) 52/1032 (5.0%) 51/1032 (4.9%) 42/1032 (4.1%) 29/1032 (2.8%) 35/1032 (3.4%) 31/1032 (3.0%) 21/1032 (2.0%) 25/1032 (2.4%) 12/1032 (1.2%) 3/1032 (0.3%) 4/1032 (0.4%)
0.10 0.96 0.06 0.97 0.05 0.98 0.09 0.97 0.05 0.98 0.05 0.98 0.06 0.99 0.04 1.00 0.03 0.99
Scans met TH (%)
0.13 0.93 0.09 0.95 0.10 0.94
SENSc SPECd
2/12 (16.7%) 0/3 (0.0%) 1/4 (25.0%)
5/31 (16.1%) 2/21 (9.5%) 4/25 (16.0%)
5/42 (11.9%) 4/29 (13.8%) 5/35 (14.3%)
6/71 (8.5%) 7/52 (13.5%) 5/51 (9.8%)
Met TH
42/1020 (4.1%) 44/1029 (4.3%) 43/1028 (4.2%)
39/1001 (3.9%) 42/1011 (4.2%) 40/1007 (4.0%)
39/990 (3.9%) 40/1003 (4.0%) 39/997 (3.9%)
38/961 (4.0%) 37/980 (3.8%) 39/981 (4.0%)
Did not meet TH
Proportion with a relapse in 28 days
Secondary progressive cohort
7.64
0.00
4.66
4.60
2.43
4.74
4.09
3.85
3.30
2.63
3.96
2.24
Crude ORa
2.94 (0.4–21.3) 0.00 NAg 2.53 (0.2–37.2)
2.41 (0.7–8.6) 0.73 (0.1–4.3) 1.65 (0.4–7.0)
1.58 (0.5–5.4) 1.48 (0.4–5.7) 1.51 (0.4–5.5)
1.03 (0.3–3.3) 1.73 (0.6–5.1) 1 .02 (0.3–3.5)
Adjusted OR (CI)b
0.05 0.99 0.00 1.00 0.02 1.00
0.11 0.97 0.05 0.98 0.09 0.98
0.11 0.96 0.09 0.97 0.11 0.97
0.14 0.93 0.16 0.95 0.11 0.95
SENSc SPECd
CI: confidence interval; G: guidelines (G1: threshold relative to the baseline magnetic resonance imaging (MRI) scan, G2: threshold relative to the most recent prior MRI scan, G3: threshold relative to the average of the two prior MRIs); OR: odds ratio. aCrude ORs are based on the 2×2 table of observed frequencies. bAdjusted ORs are based on the final mixed-effects logistic model which included adjustment for: Relapsing (R) cohort: time of visit; secondary progressive (SP) cohort: baseline contrast enhancing lesion (CEL), number of pre-study relapses and SP duration. cSENS: sensitivity (true positive rate), number of true positives/(number of true positives + number of false negatives). dSPEC: specificity (true negative rate), number of true negatives/(number of true negatives + number of false positives). eORs with 0.05
