NIH Public Access Author Manuscript Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
NIH-PA Author Manuscript
Published in final edited form as: Grand rounds Urol. 2009 August ; 8(3): 7–13.
New Prognostic Markers: The Pathway from Research to Clinical Practice Caroline Savage and Andrew J. Vickers Department of Epidemiology and Biostatistics Memorial Sloan Kettering Cancer Center
Introduction
NIH-PA Author Manuscript
The use of markers to identify cancer stretches back more than 150 years. In 1847, Sir Henry Bence-Jones discovered a protein in urine that became slightly opaque when cooled. That protein, later identified as the light chain of immunoglobulin G, served as the definitive diagnostic test for myeloma for over 100 years1. Recent advances in such areas as geneexpression microarrays, proteomics and immunology have uncovered a wealth of new markers putatively associated with cancer outcomes. In genitourinary oncology, tumor markers are widely used for diagnosis, prognosis, evaluation of response to therapy, and early detection of recurrence. For example, alphafetoprotein (AFP), human chorionic gonadotropic (hCG), and lactate dehydrogenase (LDH) are critical in the management of testicular cancers. The levels of these tumor markers are used in addition to the standard TNM American Joint Committee on Cancer (AJCC) classification system to determine the stage of the disease, assess prognosis and guide treatment decisions. For example, persistentmarker elevation after treatment can indicate residual cancer and the possible need for further therapy2.
NIH-PA Author Manuscript
In the case of prostate cancer, prostate-specific antigen (PSA) has revolutionized detection, monitoring and treatment. It is the only tumor marker widely used for population based screening for any cancer. In addition to detection, PSA levels provide a useful adjunct in the evaluation of therapeutic responses and signals recurrent disease before it can be detected by other diagnostic procedures. Despite their frequent use in diagnosis and prediction, none of the currently used tumor markers are perfect. Elevated AFP levels are not specific to testicular cancer and can indicate either malignant or benign liver disease3-5. Similarly, although PSA is highly specific to the prostate gland, elevated PSA is not specific to prostate cancer, and can result from benign prostatic disease. Indeed, only about 25% of men with an elevated PSA have biopsy detectable prostate cancer, resulting in an estimated 750,000 unnecessary biopsies in US men each year6. On the other hand, some have argued that PSA is insufficiently sensitive, as many men with low PSA nonetheless have prostate cancer on biopsy: in the Prostate Cancer Prevention Trial, where men were biopsied irrespective of PSA level, more
Corresponding author: Andrew Vickers Department of Epidemiology and Biostatistics Memorial Sloan Kettering Cancer Center 307 E63rd St New York, NY 10065 Phone: 646-735-8142 Fax: 646-735-0011 [email protected].
Savage and Vickers
Page 2
than 11% of men with a PSA less than 1ng/ml were found to have prostate cancer on biopsy7.
NIH-PA Author Manuscript
The suboptimal performance of many current markers, and the lack of proven markers for many urologic applications, such as the surveillance of bladder cancer or the detection and diagnosis of kidney cancer, has led to an intensive search for new markers. An estimated 4,000 papers published each year on cancer-related markers8. A simple Medline search of “prostate cancer marker” identifies an overwhelming assortment of potential new molecular markers: searching just a single month's Medline update we found papers on SALL49, polymorphisms of cytochrome P450 2B610, PCA311, EPCA-212, SOD2 gene13, PDE4D14, WAVE315, Mel-1816 and MSMB variants17. Yet despite such intense research activity, relatively few molecular markers have been successfully integrated into clinical practice.
NIH-PA Author Manuscript
One possible reason why research on markers has yet to be translated to the clinic is that marker research has not had a clear goal: in our view, investigators have not been clear on what it is we need to know about a marker in order to justify its clinical use. We propose a simple formulation: before we use a marker in clinical practice, we need to know whether it will improve clinical outcome. In other words, we need evidence that measuring the marker would change the decision a doctor would have made in the absence of the marker, and that this changed decision will benefit the patient. This formulation is comparable to our view of drug research: we use a drug because we believe it will improve patient outcome. One obvious difference between drugs and markers is that whereas drugs have a widely accepted development path – Phase I, II and III trials the sort of studies required for marker development, and the order in which they should be conducted, is less well established. Frameworks for the various phases of molecular marker development have been proposed18, but have yet to gain wide currency. We propose that the different stages of marker development can best be understood in terms of the scientific question asked at each stage (see table 1)
Stages of Molecular marker Development Can the marker be measured accurately and reproducibly?
NIH-PA Author Manuscript
After the discovery of a possible marker, one must determine whether the marker can be measured reliably. For example, researchers are beginning to use magnetic resonance methods to detect metabolites thought to be indicative of cancerous lesions 19. In order to refine these methods, studies investigate whether multiple runs of the same sample give consistent results and whether different machines give the same metabolomic spectra from the same sample. These tests measure reproducibility. Other tests evaluate the accuracy of the new method by determining whether the results from the new technique corroborate with those from other established techniques. If an assay failed to give reproducible results, it would clearly be of little clinical value and would not be worthy of subsequent research.
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 3
Does the marker distinguish between convenience samples of clearly distinguishable groups of patients?
NIH-PA Author Manuscript
In the early phases of marker assessment, investigators often use convenience samples of currently available specimens from clearly distinguishable groups of patients. These studies are relatively quick and easy to conduct, and allow one to screen for promising markers that warrant further study. For example, a study might compare the marker levels in the tissues of men with advanced prostate cancer to those in young men known to be free of prostate cancer or women with breast cancer20. If the marker failed to distinguish advanced disease from normal controls it would clearly not merit further study.
NIH-PA Author Manuscript
Often these studies will report a measure of diagnostic accuracy, such as sensitivity or specificity. Sensitivity is the percentage of patients with the disease who receive a positive test result; specificity is the percentage of healthy individuals who test negative. The sensitivity and specificity reported from studies using convenience samples are highly dependent on study design and are often overly optimistic when considering a new marker's use in the clinic21. It is intuitive that it would be easier to distinguish men with metastatic prostate cancer from healthy controls than it would be to distinguish men with localized cancer from patients with benign prostate disease. To illustrate how this point impacts sensitivity and specificity as reported in a study, consider a marker that distinguished advanced cancer from healthy controls with high sensitivity and specificity (both 90%), but did not classify patients with localized cancer differently from those with a benign condition (sensitivity and specificity of 50%). The results from two different studies are shown in Table 2. When the convenience samples include an equal number of patients in each group, the overall sensitivity and specificity are given as 70% (Table 2A); however, when the number of patients in each group more accurately reflects the distribution of patients likely seen in the clinic (Table 2B) - where the majority of patients have a benign condition, and very few have advanced disease - the sensitivity and specificity are much lower (in this example, 58% and 63% respectively).
NIH-PA Author Manuscript
A demonstration that levels of a marker differ between clearly dissimilar patients can be a useful step in the development of any marker. However, because diagnostic test characteristics such as sensitivity and specific cannot be assessed in such studies, the marker's performance needs to be assessed in patients reflective of those to whom it would be applied in practice. Is the marker associated with outcome in the sort of patients to whom the marker would be applied to in practice? Before using a drug to treat stage IV renal cancer, we generally like to see a clinical trial of the drug in patients with stage IV renal cancer; a trial of neoadjuvant therapy for prostate cancer should be conducted in a radical prostatectomy population. By the same token, before using a marker in the clinic, we need to determine whether the marker is associated with outcome in the sort of patients to whom the marker would be applied in practice. In the case of a marker to detect prostate cancer, for example, we would look at whether a new marker predicted biopsy outcome in a group of men undergoing biopsy22. If the marker could not
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 4
predict biopsy result in a typical sample of men referred for biopsy, then it could not be of clinical use and would not warrant further investigation.
NIH-PA Author Manuscript
That said, while demonstrating an association between the marker and outcome is an important step, it is not sufficient to determine whether a marker should be incorporated into clinical practice. This is because the new marker may not provide any information that is not already reflected by known prognostic factors. Does the marker provide additional information to that already available to the clinician? Routine clinical data, such as the patient's age and health status, and the stage and grade of the tumor, are often of predictive value. Any new prognostic maker is therefore only worth measuring if it provides information additional to these routinely available predictors. As such, an important step in evaluating the value of a new marker is to compare the predictive accuracy of a model including only standard clinical variables with that of a model including standard clinical variables plus the new marker23, 24.
NIH-PA Author Manuscript
Commonly reported measures of predictive ability are the area under the receiver operating curve (AUC) and the concordance index (C-index). Both metrics measure discrimination, that is, the ability of the model to correctly classify individuals with and without the disease. They summarize the probability that, given two randomly selected patients, the patient predicted to have the worse outcome does in fact have the worse outcome. They range from 0.5 (the model is no better than a coin-flip) to 1.0 (perfect ability to predict or rank outcome)25. As an example, Margulis et al. investigated whether the addition of Ki-67 could improve the accuracy of a model to predict recurrence or disease-specific survival after radical cystectomy in 713 patients with advanced bladder cancer26. They first calculated the predictive accuracy of the standard multivariable model, which included tumor stage, grade, nodal status, and lymphovascular invasion. They then repeated their analysis including Ki-67 in the model along with the other predictors. Addition of Ki-67 improved accuracy by 2.9% for disease recurrence and 2.4% for bladder cancer-specific survival. Thus assessment of Ki-67 expression provides the clinician with information not captured by established predictors.
NIH-PA Author Manuscript
That said, accuracy is not synonymous with clinical value: it is unclear whether an increase in predictive accuracy would translate into improved patient outcomes. One may find, for example, that Ki-67 is only able to reclassify some “high risk” patients as “very high risk.” If all patients who are high risk or above are given the same treatment regimen, use of Ki-67 would not change patient management and thus not improve outcomes. Thus an improvement in the predictive accuracy, while necessary, is not sufficient to assess whether using the marker in practice would actually benefit patients. Does the use of the marker improve clinical outcome? As discussed in the introduction, use of a marker in the clinic is only warranted if it can be shown that its use would lead to improved health outcomes. An experimental assessment, such as the randomization of patients to clinical management that does or does not
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 5
NIH-PA Author Manuscript
incorporate information from the marker, provides the most reliable data on the clinical value of a new marker. The use of PSA for cancer detection was recently subjected to randomized trial in independent trials in Europe27 and the United States28. The substantial human and financial resources necessary for randomized trials make such an evaluation impractical for most markers. Decision analytic techniques can often be used in lieu of experimental studies. The key point of decision analysis is that the consequences of clinical decisions are incorporated in analyses. For example, a patient with an elevated PSA who must decide whether they will undergo a biopsy is faced with four possible outcomes: finding cancer (true positive), an unnecessary biopsy (false positive), missing cancer (false negative), and avoiding an unnecessary biopsy (true negative). The benefit of finding a cancer early is clearly different than the harm of an unnecessary biopsy. Decision analysis allows one to weight the relative value of the benefits (true positives) to the harms (false positives) and thus incorporate the consequences of a clinical decision.
NIH-PA Author Manuscript
For example, imagine that we have data from a study of 1,000 men with elevated PSA, all of whom had a molecular marker measured before biopsy. We put this marker alongside PSA in a statistical model to predict biopsy outcome and use a risk threshold of 20% for “high risk”: were we to use the statistical model in practice, we would biopsy any man with a risk of 20% or more, and advise against biopsy for men with less than a 20% risk. Let us say that there were a total of 300 men with cancer on biopsy, and that of 510 men with a 20% or greater predicted risk, 210 had cancer. These results are shown in table 3. The question is whether it is better to biopsy 1000 men and find 300 cancers, or biopsy 510 men to find 210 cancers (or to put it another way: it is worth missing 90 cancers to avoid 490 biopsies?) A simple decision analytic approach is that a risk threshold of 20% implies that finding a cancer is four times more important than avoiding an unnecessary biopsy (20% is 20:80, or 4:1). To work out the best strategy – biopsy everyone or biopsy according to the marker risk prediction model - we calculate a “net benefit”. Just like in business, “net” is income minus expenditure; in the case of prostate biopsy, income -what you gain – is finding cancer; expenditure – what you spend – is unnecessary biopsy. To calculate a net benefit, therefore, we count up cancers found and subtract unnecessary biopsies, dividing the latter by 4 to reflect the relative importance of finding cancer and avoiding biopsy.
NIH-PA Author Manuscript
The calculation of net benefit is shown in Table 3. The marker has a higher net benefit than biopsying all patients with elevated PSA. We can therefore conclude that biopsying on the basis of the statistical model incorporating the marker would lead to better clinical outcomes. Naturally, some physicians may feel that the financial costs and patient discomfort associated with biopsy are such that that no more than three unnecessary biopsies should be conducted in order to detect one prostate cancer (i.e. they would only biopsy patients with at least a 25% risk); other physicians, or risk averse patients, may feel that catching a cancer is worth nine unnecessary biopsies (a risk threshold of 10%). Accordingly, methods in decision analytic techniques have been developed that can compare the clinical value of a
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 6
marker across a range of possible risk thresholds, each of which reflect different preferences about the benefits and harms of treatment29.
NIH-PA Author Manuscript
The Evaluation of Markers in Practice We have recently published a systematic review of the statistical methods used in molecular marker research in oncology. The majority of the 129 eligible papers focused on whether the marker has a statistical association with outcome. A minority incorporated multivariable modeling (36%), with very few including any measure of predictive accuracy (11%). Only a single paper compared predictive accuracy of the marker with standard clinical variables and none used decision analytic or experimental methods to determine whether the marker was of clinical value. Despite this, a large proportion of papers, 40%, gave clinical recommendations, thus failing to make the time-honored distinction between statistical and clinical significance30. To explore some of these issues more thoroughly in genitourinary oncology, we applied our schema for marker evaluation to several prominent markers.
NIH-PA Author Manuscript
EPCA-2 for prostate cancer detection Early prostate cancer antigen 2 (EPCA-2) was recently reported to be highly specific in discriminating between people with and without prostate cancer. The authors compared 63 normal, healthy men and 35 men with BPH to 80 men with prostate cancer (half with organ confined and half with non-organ confined disease) and reported that the specificity was 92%31. These results prompted considerable press coverage and claims that EPCA-2 would “help eliminate tens of thousands of unnecessary biopsies at the same time that it detects many tumors that are now missed by [PSA]”32, 33.
NIH-PA Author Manuscript
Yet no study has specifically evaluated whether the use of EPCA-2 in the clinic would lead to a reduction in biopsies. The study population used to determine the efficacy of EPCA-2 was not a representative sample of men with and without prostate cancer, and thus it is impossible to assess what the performance (sensitivity and specificity) would be in a true clinical or screening setting. Nor has it been shown whether EPCA-2 would provide any additional information above that of PSA or other prognostic variables. While EPCA-2 may prove to be a useful adjunct in prostate cancer screening or diagnosis, claims that it will improve cancer detection, or reduce the number of unnecessary biopsies, are premature. Genotypes and prostate cancer risk Single nucleotide polymorphisms (SNPs) in five chromosomal regions were recently found to be significantly associated with prostate cancer34. Although risk of prostate cancer was only moderately raise by each of the SNPs, the authors found a strong cumulative association. After adjustment for age, family history and geographic region, those who had four or more of the associated genotypes had a greater than four-fold odds of cancer. These findings have been widely reported and the authors have announced plans to market a genetic test35.
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 7
NIH-PA Author Manuscript
However, although there was a very strong evidence of an association between the genotypes and cancer – that is, the p value was very low - the genotypes have limited predictive ability: the AUC was 0.57, little better than a coin flip 36. The authors of the original paper reported that the predictive accuracy of a model including age, family history and geographic region was significantly improved by including the number of genotypes associated with prostate cancer at the five SNPs. But the authors did not evaluate whether the SNPs added predictive accuracy to PSA, which is known to be a very strong predictor of future cancer risk up to 25 years subsequently (AUC of 0.76)37. Subsequent research have found that when the SNPs were added to a predictive model that did include PSA, the AUC was only marginally improved38. This suggests that the information from the proposed genetic test would not appreciably advance our ability to predict which men actually had prostate cancer. While the association between these genetic polymorphisms and prostate cancer may have important biological implications, it is unlikely to be of value for clinical practice. Prostate Px
NIH-PA Author Manuscript
The prediction of a recurrence after radical prostatectomy is often calculated by using a “nomogram” 39, 40. These models incorporate known prognostic variables, such as PSA, Gleason grade, extracapsular extension, lymph node involvement, and surgical margins to estimate the probability of disease recurrence following prostatectomy. A “systems pathology” model has been developed that integrates additional clinicopathologic features, including multiple proteins identified by quantitative immunofluorescence and a total of 496 morphometric features identified through proprietary imaging software. However, an independent study found no evidence that systems pathology was of any practical benefit, as the models had lower predictive accuracy than the standard nomogram41. Despite this unambiguously negative result, the biotechnology company who created the technology have recently released a prognostic test, termed Prostate Px, that they claim will “enable more-informed decisions at prostate cancer diagnosis”42. The company website also states that the new test will benefit patients who are diagnosed with prostate cancer and are candidates for surgical treatment. PSA velocity
NIH-PA Author Manuscript
The modest diagnostic accuracy of PSA has led investigators to evaluate additional markers that could help predict prostate cancer. Recently, a proliferation of studies have looked at PSA velocity and prostate cancer, driven in part by strong biological plausibility: cancer is a growth process and it seems reasonable that the rate of change in a tumor marker would mirror the aggressiveness of the disease. The overwhelming majority of papers that study the association between PSA velocity and risk of prostate cancer focused solely on testing the statistical significance of this association. For example, Carter et al. reported that PSA velocity was statistically associated with a higher risk of death from prostate cancer many years later43. Studies like these have influenced clinical practice: the NCCN guidelines for the early detection of prostate cancer suggest that men with a PSA velocity greater than 0.35 ng/ml per year consider biopsy, even if their PSA is low44.
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 8
NIH-PA Author Manuscript
Yet there is a near complete lack of evidence that pretreatment PSA velocity is of clinical value. The paper cited in the NCCN guidelines does not include baseline PSA in the model; the authors claim that PSA and PSA velocity were too highly correlated to include both43. It is not surprising that a variable that is highly correlated with a known prognostic maker would itself be significantly associated with prognosis, but it also follows that the novel marker would unlikely to be of additional benefit. A systematic review of 87 studies that calculated a pretreatment PSA velocity or doubling time found only two that compared the predictive accuracy of a model including both a PSA dynamic and PSA with the accuracy of a model including PSA alone45. One of the studies found that PSA velocity added no improvement in predictive accuracy7; the other showed a minor improvement, but was subject to verification bias46. No paper used decision analytic techniques to evaluate the benefit of using PSA velocity in the clinic. The systematic review therefore concluded that there was insufficient evidence to warrant the use of PSA velocity in clinical practice. PSA isoforms
NIH-PA Author Manuscript
The poor specificity of PSA for prostate cancer has also led investigators to study several molecular isoforms of PSA, such as free PSA and single chain (“intact”) PSA, as well as human kallikrein-related peptidase 2 (hK2), a serine protease that shares 80% sequence homology with PSA. Early work indicated that these markers were significantly associated with prostate cancer47, 48. We subsequently demonstrated that in previously unscreened men with an elevated PSA, a statistical model incorporating these markers could predict the result of biopsy more accurately than a model including total PSA and age alone: the AUC improved from 0.68 to 0.83 with the addition of free and intact PSA plus hK26. We then estimated the clinical impact of using the panel to determine biopsy and, using decision analytic techniques, estimated that a strategy of biopsying men only with a 20% or greater risk of prostate cancer from the panel is equivalent to a strategy that reduced the number of biopsies by 35% but which missed no additional cancers6. Thus these additional markers importantly enhance our ability to predict the risk of finding prostate cancer on biopsy, and critically, would lead to an improvement in patient outcomes: a substantial reduction in the number of unnecessary biopsies.
Conclusion NIH-PA Author Manuscript
This article has focused on the use of clinically relevant statistics in the evaluation of new markers. We have omitted a discussion of the numerous other potential problems that commonly plague marker studies, such as inadequate sample sizes, inflated significance levels due to multiple testing, improper cut point optimization, and issues of verification bias; these topics have been thoughtfully addressed elsewhere49-52. In sum, the search for new molecular markers has resulted in a proliferation of tumor marker studies. The development and evaluation of these markers towards clinical implementation is a multi-step process that involves a considerable investment in time and financial resources. It should be clear that the early stage research is necessary, but rarely sufficient to show that a marker is of value. Nonetheless, some markers are aggressively promoted after showing differences between convenience samples or statistical associations with cancer
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 9
NIH-PA Author Manuscript
outcome. In order for markers to be effectively incorporated into clinical practice, we must critically evaluate whether the new marker can improve our ability to predict and, most importantly, that the use of the marker would improve clinical outcome for our patients.
Acknowledgments Supported in part by funds from David H. Koch provided through the Prostate Cancer Foundation, the Sidney Kimmel Center for Prostate and Urologic Cancers and P50-CA92629 SPORE grant from the National Cancer Institute to Dr. P. T. Scardino.
References Cited
NIH-PA Author Manuscript NIH-PA Author Manuscript
1. Kahn SN. Dear Dr. [Bence] Jones. Clin Chem. Sep; 1991 37(9):1557–1558. [PubMed: 1893590] 2. Sturgeon CM, Duffy MJ, Stenman UH, et al. National Academy of Clinical Biochemistry laboratory medicine practice guidelines for use of tumor markers in testicular, prostate, colorectal, breast, and ovarian cancers. Clin Chem. Dec; 2008 54(12):e11–79. [PubMed: 19042984] 3. Zhou L, Liu J, Luo F. Serum tumor markers for detection of hepatocellular carcinoma. World J Gastroenterol. Feb 28; 2006 12(8):1175–1181. [PubMed: 16534867] 4. Chu CW, Hwang SJ, Luo JC, et al. Clinical, virologic, and pathologic significance of elevated serum alpha-fetoprotein levels in patients with chronic hepatitis C. J Clin Gastroenterol. Mar; 2001 32(3):240–244. [PubMed: 11246354] 5. Liaw YF, Tai DI, Chen TJ, Chu CM, Huang MJ. Alpha-fetoprotein changes in the course of chronic hepatitis: relation to bridging hepatic necrosis and hepatocellular carcinoma. Liver. Jun; 1986 6(3): 133–137. [PubMed: 2427909] 6. Vickers AJ, Cronin AM, Aus G, et al. A panel of kallikrein markers can reduce unnecessary biopsy for prostate cancer: data from the European Randomized Study of Prostate Cancer Screening in Goteborg, Sweden. BMC Med. 2008; 6:19. [PubMed: 18611265] 7. Thompson IM, Ankerst DP, Chi C, et al. Assessing prostate cancer risk: results from the Prostate Cancer Prevention Trial. J Natl Cancer Inst. Apr 19; 2006 98(8):529–534. [PubMed: 16622122] 8. Ludwig JA, Weinstein JN. Biomarkers in cancer staging, prognosis and treatment selection. Nat Rev Cancer. Nov; 2005 5(11):845–856. [PubMed: 16239904] 9. Cao D, Li J, Guo CC, Allan RW, Humphrey PA. SALL4 Is a Novel Diagnostic Marker for Testicular Germ Cell Tumors. Am J Surg Pathol. Apr 22.2009 10. Kurosaki T, Suzuki M, Enomoto Y, et al. Polymorphism of cytochrome P450 2B6 and prostate cancer risk: a significant association in a Japanese population. Int J Urol. Apr; 2009 16(4):364– 368. [PubMed: 19425200] 11. Hessels D, Schalken JA. The use of PCA3 in the diagnosis of prostate cancer. Nat Rev Urol. May; 2009 6(5):255–261. [PubMed: 19424173] 12. Leman ES, Magheli A, Cannon GW, Mangold L, Partin AW, Getzenberg RH. Analysis of a serum test for prostate cancer that detects a second epitope of EPCA-2. Prostate. May 5.2009 13. Hernandez-Saavedra D, McCord JM. Association of a new intronic polymorphism of the SOD2 gene (G1677T) with cancer. Cell Biochem Funct. Apr 29; 2009 27(4):223–227. [PubMed: 19405048] 14. Rahrmann EP, Collier LS, Knutson TP, et al. Identification of PDE4D as a proliferation promoting factor in prostate cancer using a Sleeping Beauty transposon-based somatic mutagenesis screen. Cancer Res. May 15; 2009 69(10):4388–4397. [PubMed: 19401450] 15. Fernando HS, Sanders AJ, Kynaston HG, Jiang WG. WAVE3 is associated with invasiveness in prostate cancer cells. Urol Oncol. Apr 21.2009 16. Wang W, Lin T, Huang J, Hu W, Xu K, Liu J. Analysis of Mel-18 expression in prostate cancer tissues and correlation with clinicopathologic features. Urol Oncol. Apr 21.2009 17. Lou H, Yeager M, Li H, et al. Fine mapping and functional analysis of a common variant in MSMB on chromosome 10q11.2 associated with prostate cancer susceptibility. Proc Natl Acad Sci U S A. May 12; 2009 106(19):7933–7938. [PubMed: 19383797]
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 10
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript
18. Pepe MS, Etzioni R, Feng Z, et al. Phases of biomarker development for early detection of cancer. J Natl Cancer Inst. Jul 18; 2001 93(14):1054–1061. [PubMed: 11459866] 19. Sreekumar A, Poisson LM, Rajendiran TM, et al. Metabolomic profiles delineate potential role for sarcosine in prostate cancer progression. Nature. Feb 12; 2009 457(7231):910–914. [PubMed: 19212411] 20. Beckett ML, Cazares LH, Vlahou A, Schellhammer PF, Wright GL Jr. Prostate-specific membrane antigen levels in sera from healthy men and patients with benign prostate hyperplasia or prostate cancer. Clin Cancer Res. Dec; 1999 5(12):4034–4040. [PubMed: 10632336] 21. Ransohoff DF, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med. Oct 26; 1978 299(17):926–930. [PubMed: 692598] 22. Steuber T, Vickers A, Haese A, et al. Free PSA isoforms and intact and cleaved forms of urokinase plasminogen activator receptor in serum improve selection of patients for prostate cancer biopsy. Int J Cancer. Apr 1; 2007 120(7):1499–1504. [PubMed: 17205511] 23. Katz EM, Kattan MW. How to judge a tumor marker. Nat Clin Pract Oncol. Oct; 2005 2(10):482– 483. [PubMed: 16205748] 24. Kattan MW. Judging new markers by their ability to improve predictive accuracy. J Natl Cancer Inst. May 7; 2003 95(9):634–635. [PubMed: 12734304] 25. Harrell, FE. Regression Modelling Strategies. Springer-Verlag; New York: 2001. 26. Margulis V, Lotan Y, Karakiewicz PI, et al. Multi-institutional validation of the predictive value of Ki-67 labeling index in patients with urinary bladder cancer. J Natl Cancer Inst. Jan 21; 2009 101(2):114–119. [PubMed: 19141773] 27. Schroder FH, Hugosson J, Roobol MJ, et al. Screening and prostate-cancer mortality in a randomized European study. N Engl J Med. Mar 26; 2009 360(13):1320–1328. [PubMed: 19297566] 28. Andriole GL, Crawford ED, Grubb RL 3rd, et al. Mortality results from a randomized prostatecancer screening trial. N Engl J Med. Mar 26; 2009 360(13):1310–1319. [PubMed: 19297565] 29. Vickers AJ. Decision analysis for the evaluation of diagnostic tests, prediction models and molecular markers. Am Stat. 2008; 62(4):314–320. [PubMed: 19132141] 30. Vickers AJ, Jang K, Sargent D, Lilja H, Kattan MW. Systematic review of statistical methods used in molecular marker studies in cancer. Cancer. Apr 15; 2008 112(8):1862–1868. [PubMed: 18320601] 31. Leman ES, Cannon GW, Trock BJ, et al. EPCA-2: a highly specific serum marker for prostate cancer. Urology. Apr; 2007 69(4):714–720. [PubMed: 17445657] 32. New Prostate Cancer Test May Detect More Tumors. The Washington Post. Apr.2007 26 33. Szabo L. Test shows promise in detecting prostate cancer. USA Today. Apr 26.2007 34. Zheng SL, Sun J, Wiklund F, et al. Cumulative association of five genetic variants with prostate cancer. N Engl J Med. Feb 28; 2008 358(9):910–919. [PubMed: 18199855] 35. Kolata G. $300 to Learn RIsk of Prostate Cancer. The New York Times. Jan 17.2008 36. Vickers A, Lilja H, Scardino P. Five genetic variants associated with prostate cancer. N Engl J Med. Jun 19.2008 358(25):2740. author reply 2741. [PubMed: 18572449] 37. Lilja H, Ulmert D, Bjork T, et al. Long-term prediction of prostate cancer up to 25 years before diagnosis of prostate cancer using prostate kallikreins measured at age 44 to 50 years. J Clin Oncol. Feb 1; 2007 25(4):431–436. [PubMed: 17264339] 38. Salinas CA, Koopmeiners JS, Kwon EM, et al. Clinical utility of five genetic variants for predicting prostate cancer risk and mortality. Prostate. Mar 1; 2009 69(4):363–372. [PubMed: 19058137] 39. Kattan MW, Wheeler TM, Scardino PT. Postoperative nomogram for disease recurrence after radical prostatectomy for prostate cancer. J Clin Oncol. May; 1999 17(5):1499–1507. [PubMed: 10334537] 40. Stephenson AJ, Scardino PT, Eastham JA, et al. Postoperative nomogram predicting the 10-year probability of prostate cancer recurrence after radical prostatectomy. J Clin Oncol. Oct 1; 2005 23(28):7005–7012. [PubMed: 16192588]
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 11
NIH-PA Author Manuscript NIH-PA Author Manuscript
41. Eggener SE, Vickers AJ, Serio AM, et al. Comparison of models to predict clinical failure after radical prostatectomy. Cancer. Jan 15; 2009 115(2):303–310. [PubMed: 19025977] 42. Aureon Laboratories. Prognostic Tests. Prostate Px. http://aureon.com/prognostic-tests-prostatepx.htm 43. Carter HB, Ferrucci L, Kettermann A, et al. Detection of life-threatening prostate cancer with prostate-specific antigen velocity during a window of curability. J Natl Cancer Inst. Nov 1; 2006 98(21):1521–1527. [PubMed: 17077354] 44. Prostate cancer detection (Version 2.2007), National Comprehensive Cancer Network. 2007. (www.nccn.org) 45. Vickers AJ, Savage C, O'Brien MF, Lilja H. Systematic review of pretreatment prostate-specific antigen velocity and doubling time as predictors for prostate cancer. J Clin Oncol. Jan 20; 2009 27(3):398–403. [PubMed: 19064972] 46. Loeb S, Roehl KA, Catalona WJ, Nadler RB. Prostate specific antigen velocity threshold for predicting prostate cancer in young men. J Urol. Mar; 2007 177(3):899–902. [PubMed: 17296371] 47. Roddam AW, Duffy MJ, Hamdy FC, et al. Use of prostate-specific antigen (PSA) isoforms for the detection of prostate cancer in men with a PSA level of 2-10 ng/ml: systematic review and metaanalysis. Eur Urol. Sep; 2005 48(3):386–399. discussion 398-389. [PubMed: 15982797] 48. Becker C, Piironen T, Pettersson K, Hugosson J, Lilja H. Clinical value of human glandular kallikrein 2 and free and total prostate-specific antigen in serum from a population of men with prostate-specific antigen levels 3.0 ng/mL or greater. Urology. May; 2000 55(5):694–699. [PubMed: 10792083] 49. Pajak TF, Clark GM, Sargent DJ, McShane LM, Hammond ME. Statistical issues in tumor marker studies. Arch Pathol Lab Med. Jul; 2000 124(7):1011–1015. [PubMed: 10888777] 50. Cronin AM, Vickers AJ. Statistical methods to correct for verification bias in diagnostic studies are inadequate when there are few false negatives: a simulation study. BMC Med Res Methodol. 2008; 8:75. [PubMed: 19014457] 51. Etzioni R. Statistical issues in the evaluation of screening and early detection modalities. Urol Oncol. May-Jun;2008 26(3):308–315. [PubMed: 18452826] 52. McShane LM, Altman DG, Sauerbrei W, et al. Reporting recommendations for tumor marker prognostic studies (REMARK). J Natl Cancer Inst. Aug 17; 2005 97(16):1180–1184. [PubMed: 16106022]
NIH-PA Author Manuscript Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 12
Table 1
NIH-PA Author Manuscript
Stages of marker development 1.
Can the marker be measured accurately and reproducibly?
2.
Does the marker distinguish between convenience samples of clearly distinguishable groups of patients?
3.
Is the marker associated with outcome in the sort of patients to whom the marker would be applied to in practice?
4.
Does the marker provide additional information to that already available to the clinician?
5.
Does the use of the marker improve clinical outcome?
NIH-PA Author Manuscript NIH-PA Author Manuscript Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 13
Table 2
NIH-PA Author Manuscript
Two hypothetical studies that use convenience samples of subjects without disease, benign disease, localized, or advanced cancer The marker's sensitivity and specificity for advanced disease and those without disease is high (both 90%); however, the marker is unable to correctly patients with localized disease or those with a benign condition (sensitivity and specificity are 50%). A) Equal numbers of patients are included in each group (healthy, benign, local cancer and advanced cancer). The overall sensitivity and specificity are both 70%. B) Sampling of patients reflects the prevalence of groups seen in the clinic. In this example 25% of patients have no disease, 50% have a benign condition, 20% of patients have localized cancer and 5% have advanced cancer. The sensitivity is 58% and the specificity is 63%. A) Cancer (Advanced or Localized)
No Cancer (No disease or benign condition)
Marker +
90% × 500 + 50% × 500 = 700 (True positives)
10% × 500 + 50% × 500 = 300 (False positives)
Marker -
10% × 500 + 50% × 500 = 300 (False negatives)
90% × 500 + 50% × 500 = 700 (True negatives)
1000
1000
NIH-PA Author Manuscript
B)
Marker + Marker -
Cancer (Advanced or Localized)
No Cancer (No disease or benign condition)
90% × 50 + 50% × 200 = 145 (True positives)
10% × 250 + 50% × 500 = 275 (False positives)
10% × 50 + 50% × 200 = 105 (False negatives)
90% × 250 + 50% × 500 = 475 (True negatives)
250
750
NIH-PA Author Manuscript Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 14
Table 3
NIH-PA Author Manuscript
Hypothetical results of a marker study for prostate biopsy, showing a decision analysis. Strategy
Cancers found
Unnecessary Biopsies
Net benefit
Biopsy all men with elevated PSA
300
700
300 - 700 ÷ 4=125
Biopsy if risk ≥ 20% from statistical model based on molecular marker
210
300
210 - 300 ÷ 4=135
NIH-PA Author Manuscript NIH-PA Author Manuscript Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
NIH-PA Author Manuscript
Published in final edited form as: Grand rounds Urol. 2009 August ; 8(3): 7–13.
New Prognostic Markers: The Pathway from Research to Clinical Practice Caroline Savage and Andrew J. Vickers Department of Epidemiology and Biostatistics Memorial Sloan Kettering Cancer Center
Introduction
NIH-PA Author Manuscript
The use of markers to identify cancer stretches back more than 150 years. In 1847, Sir Henry Bence-Jones discovered a protein in urine that became slightly opaque when cooled. That protein, later identified as the light chain of immunoglobulin G, served as the definitive diagnostic test for myeloma for over 100 years1. Recent advances in such areas as geneexpression microarrays, proteomics and immunology have uncovered a wealth of new markers putatively associated with cancer outcomes. In genitourinary oncology, tumor markers are widely used for diagnosis, prognosis, evaluation of response to therapy, and early detection of recurrence. For example, alphafetoprotein (AFP), human chorionic gonadotropic (hCG), and lactate dehydrogenase (LDH) are critical in the management of testicular cancers. The levels of these tumor markers are used in addition to the standard TNM American Joint Committee on Cancer (AJCC) classification system to determine the stage of the disease, assess prognosis and guide treatment decisions. For example, persistentmarker elevation after treatment can indicate residual cancer and the possible need for further therapy2.
NIH-PA Author Manuscript
In the case of prostate cancer, prostate-specific antigen (PSA) has revolutionized detection, monitoring and treatment. It is the only tumor marker widely used for population based screening for any cancer. In addition to detection, PSA levels provide a useful adjunct in the evaluation of therapeutic responses and signals recurrent disease before it can be detected by other diagnostic procedures. Despite their frequent use in diagnosis and prediction, none of the currently used tumor markers are perfect. Elevated AFP levels are not specific to testicular cancer and can indicate either malignant or benign liver disease3-5. Similarly, although PSA is highly specific to the prostate gland, elevated PSA is not specific to prostate cancer, and can result from benign prostatic disease. Indeed, only about 25% of men with an elevated PSA have biopsy detectable prostate cancer, resulting in an estimated 750,000 unnecessary biopsies in US men each year6. On the other hand, some have argued that PSA is insufficiently sensitive, as many men with low PSA nonetheless have prostate cancer on biopsy: in the Prostate Cancer Prevention Trial, where men were biopsied irrespective of PSA level, more
Corresponding author: Andrew Vickers Department of Epidemiology and Biostatistics Memorial Sloan Kettering Cancer Center 307 E63rd St New York, NY 10065 Phone: 646-735-8142 Fax: 646-735-0011 [email protected].
Savage and Vickers
Page 2
than 11% of men with a PSA less than 1ng/ml were found to have prostate cancer on biopsy7.
NIH-PA Author Manuscript
The suboptimal performance of many current markers, and the lack of proven markers for many urologic applications, such as the surveillance of bladder cancer or the detection and diagnosis of kidney cancer, has led to an intensive search for new markers. An estimated 4,000 papers published each year on cancer-related markers8. A simple Medline search of “prostate cancer marker” identifies an overwhelming assortment of potential new molecular markers: searching just a single month's Medline update we found papers on SALL49, polymorphisms of cytochrome P450 2B610, PCA311, EPCA-212, SOD2 gene13, PDE4D14, WAVE315, Mel-1816 and MSMB variants17. Yet despite such intense research activity, relatively few molecular markers have been successfully integrated into clinical practice.
NIH-PA Author Manuscript
One possible reason why research on markers has yet to be translated to the clinic is that marker research has not had a clear goal: in our view, investigators have not been clear on what it is we need to know about a marker in order to justify its clinical use. We propose a simple formulation: before we use a marker in clinical practice, we need to know whether it will improve clinical outcome. In other words, we need evidence that measuring the marker would change the decision a doctor would have made in the absence of the marker, and that this changed decision will benefit the patient. This formulation is comparable to our view of drug research: we use a drug because we believe it will improve patient outcome. One obvious difference between drugs and markers is that whereas drugs have a widely accepted development path – Phase I, II and III trials the sort of studies required for marker development, and the order in which they should be conducted, is less well established. Frameworks for the various phases of molecular marker development have been proposed18, but have yet to gain wide currency. We propose that the different stages of marker development can best be understood in terms of the scientific question asked at each stage (see table 1)
Stages of Molecular marker Development Can the marker be measured accurately and reproducibly?
NIH-PA Author Manuscript
After the discovery of a possible marker, one must determine whether the marker can be measured reliably. For example, researchers are beginning to use magnetic resonance methods to detect metabolites thought to be indicative of cancerous lesions 19. In order to refine these methods, studies investigate whether multiple runs of the same sample give consistent results and whether different machines give the same metabolomic spectra from the same sample. These tests measure reproducibility. Other tests evaluate the accuracy of the new method by determining whether the results from the new technique corroborate with those from other established techniques. If an assay failed to give reproducible results, it would clearly be of little clinical value and would not be worthy of subsequent research.
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 3
Does the marker distinguish between convenience samples of clearly distinguishable groups of patients?
NIH-PA Author Manuscript
In the early phases of marker assessment, investigators often use convenience samples of currently available specimens from clearly distinguishable groups of patients. These studies are relatively quick and easy to conduct, and allow one to screen for promising markers that warrant further study. For example, a study might compare the marker levels in the tissues of men with advanced prostate cancer to those in young men known to be free of prostate cancer or women with breast cancer20. If the marker failed to distinguish advanced disease from normal controls it would clearly not merit further study.
NIH-PA Author Manuscript
Often these studies will report a measure of diagnostic accuracy, such as sensitivity or specificity. Sensitivity is the percentage of patients with the disease who receive a positive test result; specificity is the percentage of healthy individuals who test negative. The sensitivity and specificity reported from studies using convenience samples are highly dependent on study design and are often overly optimistic when considering a new marker's use in the clinic21. It is intuitive that it would be easier to distinguish men with metastatic prostate cancer from healthy controls than it would be to distinguish men with localized cancer from patients with benign prostate disease. To illustrate how this point impacts sensitivity and specificity as reported in a study, consider a marker that distinguished advanced cancer from healthy controls with high sensitivity and specificity (both 90%), but did not classify patients with localized cancer differently from those with a benign condition (sensitivity and specificity of 50%). The results from two different studies are shown in Table 2. When the convenience samples include an equal number of patients in each group, the overall sensitivity and specificity are given as 70% (Table 2A); however, when the number of patients in each group more accurately reflects the distribution of patients likely seen in the clinic (Table 2B) - where the majority of patients have a benign condition, and very few have advanced disease - the sensitivity and specificity are much lower (in this example, 58% and 63% respectively).
NIH-PA Author Manuscript
A demonstration that levels of a marker differ between clearly dissimilar patients can be a useful step in the development of any marker. However, because diagnostic test characteristics such as sensitivity and specific cannot be assessed in such studies, the marker's performance needs to be assessed in patients reflective of those to whom it would be applied in practice. Is the marker associated with outcome in the sort of patients to whom the marker would be applied to in practice? Before using a drug to treat stage IV renal cancer, we generally like to see a clinical trial of the drug in patients with stage IV renal cancer; a trial of neoadjuvant therapy for prostate cancer should be conducted in a radical prostatectomy population. By the same token, before using a marker in the clinic, we need to determine whether the marker is associated with outcome in the sort of patients to whom the marker would be applied in practice. In the case of a marker to detect prostate cancer, for example, we would look at whether a new marker predicted biopsy outcome in a group of men undergoing biopsy22. If the marker could not
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 4
predict biopsy result in a typical sample of men referred for biopsy, then it could not be of clinical use and would not warrant further investigation.
NIH-PA Author Manuscript
That said, while demonstrating an association between the marker and outcome is an important step, it is not sufficient to determine whether a marker should be incorporated into clinical practice. This is because the new marker may not provide any information that is not already reflected by known prognostic factors. Does the marker provide additional information to that already available to the clinician? Routine clinical data, such as the patient's age and health status, and the stage and grade of the tumor, are often of predictive value. Any new prognostic maker is therefore only worth measuring if it provides information additional to these routinely available predictors. As such, an important step in evaluating the value of a new marker is to compare the predictive accuracy of a model including only standard clinical variables with that of a model including standard clinical variables plus the new marker23, 24.
NIH-PA Author Manuscript
Commonly reported measures of predictive ability are the area under the receiver operating curve (AUC) and the concordance index (C-index). Both metrics measure discrimination, that is, the ability of the model to correctly classify individuals with and without the disease. They summarize the probability that, given two randomly selected patients, the patient predicted to have the worse outcome does in fact have the worse outcome. They range from 0.5 (the model is no better than a coin-flip) to 1.0 (perfect ability to predict or rank outcome)25. As an example, Margulis et al. investigated whether the addition of Ki-67 could improve the accuracy of a model to predict recurrence or disease-specific survival after radical cystectomy in 713 patients with advanced bladder cancer26. They first calculated the predictive accuracy of the standard multivariable model, which included tumor stage, grade, nodal status, and lymphovascular invasion. They then repeated their analysis including Ki-67 in the model along with the other predictors. Addition of Ki-67 improved accuracy by 2.9% for disease recurrence and 2.4% for bladder cancer-specific survival. Thus assessment of Ki-67 expression provides the clinician with information not captured by established predictors.
NIH-PA Author Manuscript
That said, accuracy is not synonymous with clinical value: it is unclear whether an increase in predictive accuracy would translate into improved patient outcomes. One may find, for example, that Ki-67 is only able to reclassify some “high risk” patients as “very high risk.” If all patients who are high risk or above are given the same treatment regimen, use of Ki-67 would not change patient management and thus not improve outcomes. Thus an improvement in the predictive accuracy, while necessary, is not sufficient to assess whether using the marker in practice would actually benefit patients. Does the use of the marker improve clinical outcome? As discussed in the introduction, use of a marker in the clinic is only warranted if it can be shown that its use would lead to improved health outcomes. An experimental assessment, such as the randomization of patients to clinical management that does or does not
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 5
NIH-PA Author Manuscript
incorporate information from the marker, provides the most reliable data on the clinical value of a new marker. The use of PSA for cancer detection was recently subjected to randomized trial in independent trials in Europe27 and the United States28. The substantial human and financial resources necessary for randomized trials make such an evaluation impractical for most markers. Decision analytic techniques can often be used in lieu of experimental studies. The key point of decision analysis is that the consequences of clinical decisions are incorporated in analyses. For example, a patient with an elevated PSA who must decide whether they will undergo a biopsy is faced with four possible outcomes: finding cancer (true positive), an unnecessary biopsy (false positive), missing cancer (false negative), and avoiding an unnecessary biopsy (true negative). The benefit of finding a cancer early is clearly different than the harm of an unnecessary biopsy. Decision analysis allows one to weight the relative value of the benefits (true positives) to the harms (false positives) and thus incorporate the consequences of a clinical decision.
NIH-PA Author Manuscript
For example, imagine that we have data from a study of 1,000 men with elevated PSA, all of whom had a molecular marker measured before biopsy. We put this marker alongside PSA in a statistical model to predict biopsy outcome and use a risk threshold of 20% for “high risk”: were we to use the statistical model in practice, we would biopsy any man with a risk of 20% or more, and advise against biopsy for men with less than a 20% risk. Let us say that there were a total of 300 men with cancer on biopsy, and that of 510 men with a 20% or greater predicted risk, 210 had cancer. These results are shown in table 3. The question is whether it is better to biopsy 1000 men and find 300 cancers, or biopsy 510 men to find 210 cancers (or to put it another way: it is worth missing 90 cancers to avoid 490 biopsies?) A simple decision analytic approach is that a risk threshold of 20% implies that finding a cancer is four times more important than avoiding an unnecessary biopsy (20% is 20:80, or 4:1). To work out the best strategy – biopsy everyone or biopsy according to the marker risk prediction model - we calculate a “net benefit”. Just like in business, “net” is income minus expenditure; in the case of prostate biopsy, income -what you gain – is finding cancer; expenditure – what you spend – is unnecessary biopsy. To calculate a net benefit, therefore, we count up cancers found and subtract unnecessary biopsies, dividing the latter by 4 to reflect the relative importance of finding cancer and avoiding biopsy.
NIH-PA Author Manuscript
The calculation of net benefit is shown in Table 3. The marker has a higher net benefit than biopsying all patients with elevated PSA. We can therefore conclude that biopsying on the basis of the statistical model incorporating the marker would lead to better clinical outcomes. Naturally, some physicians may feel that the financial costs and patient discomfort associated with biopsy are such that that no more than three unnecessary biopsies should be conducted in order to detect one prostate cancer (i.e. they would only biopsy patients with at least a 25% risk); other physicians, or risk averse patients, may feel that catching a cancer is worth nine unnecessary biopsies (a risk threshold of 10%). Accordingly, methods in decision analytic techniques have been developed that can compare the clinical value of a
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 6
marker across a range of possible risk thresholds, each of which reflect different preferences about the benefits and harms of treatment29.
NIH-PA Author Manuscript
The Evaluation of Markers in Practice We have recently published a systematic review of the statistical methods used in molecular marker research in oncology. The majority of the 129 eligible papers focused on whether the marker has a statistical association with outcome. A minority incorporated multivariable modeling (36%), with very few including any measure of predictive accuracy (11%). Only a single paper compared predictive accuracy of the marker with standard clinical variables and none used decision analytic or experimental methods to determine whether the marker was of clinical value. Despite this, a large proportion of papers, 40%, gave clinical recommendations, thus failing to make the time-honored distinction between statistical and clinical significance30. To explore some of these issues more thoroughly in genitourinary oncology, we applied our schema for marker evaluation to several prominent markers.
NIH-PA Author Manuscript
EPCA-2 for prostate cancer detection Early prostate cancer antigen 2 (EPCA-2) was recently reported to be highly specific in discriminating between people with and without prostate cancer. The authors compared 63 normal, healthy men and 35 men with BPH to 80 men with prostate cancer (half with organ confined and half with non-organ confined disease) and reported that the specificity was 92%31. These results prompted considerable press coverage and claims that EPCA-2 would “help eliminate tens of thousands of unnecessary biopsies at the same time that it detects many tumors that are now missed by [PSA]”32, 33.
NIH-PA Author Manuscript
Yet no study has specifically evaluated whether the use of EPCA-2 in the clinic would lead to a reduction in biopsies. The study population used to determine the efficacy of EPCA-2 was not a representative sample of men with and without prostate cancer, and thus it is impossible to assess what the performance (sensitivity and specificity) would be in a true clinical or screening setting. Nor has it been shown whether EPCA-2 would provide any additional information above that of PSA or other prognostic variables. While EPCA-2 may prove to be a useful adjunct in prostate cancer screening or diagnosis, claims that it will improve cancer detection, or reduce the number of unnecessary biopsies, are premature. Genotypes and prostate cancer risk Single nucleotide polymorphisms (SNPs) in five chromosomal regions were recently found to be significantly associated with prostate cancer34. Although risk of prostate cancer was only moderately raise by each of the SNPs, the authors found a strong cumulative association. After adjustment for age, family history and geographic region, those who had four or more of the associated genotypes had a greater than four-fold odds of cancer. These findings have been widely reported and the authors have announced plans to market a genetic test35.
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 7
NIH-PA Author Manuscript
However, although there was a very strong evidence of an association between the genotypes and cancer – that is, the p value was very low - the genotypes have limited predictive ability: the AUC was 0.57, little better than a coin flip 36. The authors of the original paper reported that the predictive accuracy of a model including age, family history and geographic region was significantly improved by including the number of genotypes associated with prostate cancer at the five SNPs. But the authors did not evaluate whether the SNPs added predictive accuracy to PSA, which is known to be a very strong predictor of future cancer risk up to 25 years subsequently (AUC of 0.76)37. Subsequent research have found that when the SNPs were added to a predictive model that did include PSA, the AUC was only marginally improved38. This suggests that the information from the proposed genetic test would not appreciably advance our ability to predict which men actually had prostate cancer. While the association between these genetic polymorphisms and prostate cancer may have important biological implications, it is unlikely to be of value for clinical practice. Prostate Px
NIH-PA Author Manuscript
The prediction of a recurrence after radical prostatectomy is often calculated by using a “nomogram” 39, 40. These models incorporate known prognostic variables, such as PSA, Gleason grade, extracapsular extension, lymph node involvement, and surgical margins to estimate the probability of disease recurrence following prostatectomy. A “systems pathology” model has been developed that integrates additional clinicopathologic features, including multiple proteins identified by quantitative immunofluorescence and a total of 496 morphometric features identified through proprietary imaging software. However, an independent study found no evidence that systems pathology was of any practical benefit, as the models had lower predictive accuracy than the standard nomogram41. Despite this unambiguously negative result, the biotechnology company who created the technology have recently released a prognostic test, termed Prostate Px, that they claim will “enable more-informed decisions at prostate cancer diagnosis”42. The company website also states that the new test will benefit patients who are diagnosed with prostate cancer and are candidates for surgical treatment. PSA velocity
NIH-PA Author Manuscript
The modest diagnostic accuracy of PSA has led investigators to evaluate additional markers that could help predict prostate cancer. Recently, a proliferation of studies have looked at PSA velocity and prostate cancer, driven in part by strong biological plausibility: cancer is a growth process and it seems reasonable that the rate of change in a tumor marker would mirror the aggressiveness of the disease. The overwhelming majority of papers that study the association between PSA velocity and risk of prostate cancer focused solely on testing the statistical significance of this association. For example, Carter et al. reported that PSA velocity was statistically associated with a higher risk of death from prostate cancer many years later43. Studies like these have influenced clinical practice: the NCCN guidelines for the early detection of prostate cancer suggest that men with a PSA velocity greater than 0.35 ng/ml per year consider biopsy, even if their PSA is low44.
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 8
NIH-PA Author Manuscript
Yet there is a near complete lack of evidence that pretreatment PSA velocity is of clinical value. The paper cited in the NCCN guidelines does not include baseline PSA in the model; the authors claim that PSA and PSA velocity were too highly correlated to include both43. It is not surprising that a variable that is highly correlated with a known prognostic maker would itself be significantly associated with prognosis, but it also follows that the novel marker would unlikely to be of additional benefit. A systematic review of 87 studies that calculated a pretreatment PSA velocity or doubling time found only two that compared the predictive accuracy of a model including both a PSA dynamic and PSA with the accuracy of a model including PSA alone45. One of the studies found that PSA velocity added no improvement in predictive accuracy7; the other showed a minor improvement, but was subject to verification bias46. No paper used decision analytic techniques to evaluate the benefit of using PSA velocity in the clinic. The systematic review therefore concluded that there was insufficient evidence to warrant the use of PSA velocity in clinical practice. PSA isoforms
NIH-PA Author Manuscript
The poor specificity of PSA for prostate cancer has also led investigators to study several molecular isoforms of PSA, such as free PSA and single chain (“intact”) PSA, as well as human kallikrein-related peptidase 2 (hK2), a serine protease that shares 80% sequence homology with PSA. Early work indicated that these markers were significantly associated with prostate cancer47, 48. We subsequently demonstrated that in previously unscreened men with an elevated PSA, a statistical model incorporating these markers could predict the result of biopsy more accurately than a model including total PSA and age alone: the AUC improved from 0.68 to 0.83 with the addition of free and intact PSA plus hK26. We then estimated the clinical impact of using the panel to determine biopsy and, using decision analytic techniques, estimated that a strategy of biopsying men only with a 20% or greater risk of prostate cancer from the panel is equivalent to a strategy that reduced the number of biopsies by 35% but which missed no additional cancers6. Thus these additional markers importantly enhance our ability to predict the risk of finding prostate cancer on biopsy, and critically, would lead to an improvement in patient outcomes: a substantial reduction in the number of unnecessary biopsies.
Conclusion NIH-PA Author Manuscript
This article has focused on the use of clinically relevant statistics in the evaluation of new markers. We have omitted a discussion of the numerous other potential problems that commonly plague marker studies, such as inadequate sample sizes, inflated significance levels due to multiple testing, improper cut point optimization, and issues of verification bias; these topics have been thoughtfully addressed elsewhere49-52. In sum, the search for new molecular markers has resulted in a proliferation of tumor marker studies. The development and evaluation of these markers towards clinical implementation is a multi-step process that involves a considerable investment in time and financial resources. It should be clear that the early stage research is necessary, but rarely sufficient to show that a marker is of value. Nonetheless, some markers are aggressively promoted after showing differences between convenience samples or statistical associations with cancer
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 9
NIH-PA Author Manuscript
outcome. In order for markers to be effectively incorporated into clinical practice, we must critically evaluate whether the new marker can improve our ability to predict and, most importantly, that the use of the marker would improve clinical outcome for our patients.
Acknowledgments Supported in part by funds from David H. Koch provided through the Prostate Cancer Foundation, the Sidney Kimmel Center for Prostate and Urologic Cancers and P50-CA92629 SPORE grant from the National Cancer Institute to Dr. P. T. Scardino.
References Cited
NIH-PA Author Manuscript NIH-PA Author Manuscript
1. Kahn SN. Dear Dr. [Bence] Jones. Clin Chem. Sep; 1991 37(9):1557–1558. [PubMed: 1893590] 2. Sturgeon CM, Duffy MJ, Stenman UH, et al. National Academy of Clinical Biochemistry laboratory medicine practice guidelines for use of tumor markers in testicular, prostate, colorectal, breast, and ovarian cancers. Clin Chem. Dec; 2008 54(12):e11–79. [PubMed: 19042984] 3. Zhou L, Liu J, Luo F. Serum tumor markers for detection of hepatocellular carcinoma. World J Gastroenterol. Feb 28; 2006 12(8):1175–1181. [PubMed: 16534867] 4. Chu CW, Hwang SJ, Luo JC, et al. Clinical, virologic, and pathologic significance of elevated serum alpha-fetoprotein levels in patients with chronic hepatitis C. J Clin Gastroenterol. Mar; 2001 32(3):240–244. [PubMed: 11246354] 5. Liaw YF, Tai DI, Chen TJ, Chu CM, Huang MJ. Alpha-fetoprotein changes in the course of chronic hepatitis: relation to bridging hepatic necrosis and hepatocellular carcinoma. Liver. Jun; 1986 6(3): 133–137. [PubMed: 2427909] 6. Vickers AJ, Cronin AM, Aus G, et al. A panel of kallikrein markers can reduce unnecessary biopsy for prostate cancer: data from the European Randomized Study of Prostate Cancer Screening in Goteborg, Sweden. BMC Med. 2008; 6:19. [PubMed: 18611265] 7. Thompson IM, Ankerst DP, Chi C, et al. Assessing prostate cancer risk: results from the Prostate Cancer Prevention Trial. J Natl Cancer Inst. Apr 19; 2006 98(8):529–534. [PubMed: 16622122] 8. Ludwig JA, Weinstein JN. Biomarkers in cancer staging, prognosis and treatment selection. Nat Rev Cancer. Nov; 2005 5(11):845–856. [PubMed: 16239904] 9. Cao D, Li J, Guo CC, Allan RW, Humphrey PA. SALL4 Is a Novel Diagnostic Marker for Testicular Germ Cell Tumors. Am J Surg Pathol. Apr 22.2009 10. Kurosaki T, Suzuki M, Enomoto Y, et al. Polymorphism of cytochrome P450 2B6 and prostate cancer risk: a significant association in a Japanese population. Int J Urol. Apr; 2009 16(4):364– 368. [PubMed: 19425200] 11. Hessels D, Schalken JA. The use of PCA3 in the diagnosis of prostate cancer. Nat Rev Urol. May; 2009 6(5):255–261. [PubMed: 19424173] 12. Leman ES, Magheli A, Cannon GW, Mangold L, Partin AW, Getzenberg RH. Analysis of a serum test for prostate cancer that detects a second epitope of EPCA-2. Prostate. May 5.2009 13. Hernandez-Saavedra D, McCord JM. Association of a new intronic polymorphism of the SOD2 gene (G1677T) with cancer. Cell Biochem Funct. Apr 29; 2009 27(4):223–227. [PubMed: 19405048] 14. Rahrmann EP, Collier LS, Knutson TP, et al. Identification of PDE4D as a proliferation promoting factor in prostate cancer using a Sleeping Beauty transposon-based somatic mutagenesis screen. Cancer Res. May 15; 2009 69(10):4388–4397. [PubMed: 19401450] 15. Fernando HS, Sanders AJ, Kynaston HG, Jiang WG. WAVE3 is associated with invasiveness in prostate cancer cells. Urol Oncol. Apr 21.2009 16. Wang W, Lin T, Huang J, Hu W, Xu K, Liu J. Analysis of Mel-18 expression in prostate cancer tissues and correlation with clinicopathologic features. Urol Oncol. Apr 21.2009 17. Lou H, Yeager M, Li H, et al. Fine mapping and functional analysis of a common variant in MSMB on chromosome 10q11.2 associated with prostate cancer susceptibility. Proc Natl Acad Sci U S A. May 12; 2009 106(19):7933–7938. [PubMed: 19383797]
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 10
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript
18. Pepe MS, Etzioni R, Feng Z, et al. Phases of biomarker development for early detection of cancer. J Natl Cancer Inst. Jul 18; 2001 93(14):1054–1061. [PubMed: 11459866] 19. Sreekumar A, Poisson LM, Rajendiran TM, et al. Metabolomic profiles delineate potential role for sarcosine in prostate cancer progression. Nature. Feb 12; 2009 457(7231):910–914. [PubMed: 19212411] 20. Beckett ML, Cazares LH, Vlahou A, Schellhammer PF, Wright GL Jr. Prostate-specific membrane antigen levels in sera from healthy men and patients with benign prostate hyperplasia or prostate cancer. Clin Cancer Res. Dec; 1999 5(12):4034–4040. [PubMed: 10632336] 21. Ransohoff DF, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med. Oct 26; 1978 299(17):926–930. [PubMed: 692598] 22. Steuber T, Vickers A, Haese A, et al. Free PSA isoforms and intact and cleaved forms of urokinase plasminogen activator receptor in serum improve selection of patients for prostate cancer biopsy. Int J Cancer. Apr 1; 2007 120(7):1499–1504. [PubMed: 17205511] 23. Katz EM, Kattan MW. How to judge a tumor marker. Nat Clin Pract Oncol. Oct; 2005 2(10):482– 483. [PubMed: 16205748] 24. Kattan MW. Judging new markers by their ability to improve predictive accuracy. J Natl Cancer Inst. May 7; 2003 95(9):634–635. [PubMed: 12734304] 25. Harrell, FE. Regression Modelling Strategies. Springer-Verlag; New York: 2001. 26. Margulis V, Lotan Y, Karakiewicz PI, et al. Multi-institutional validation of the predictive value of Ki-67 labeling index in patients with urinary bladder cancer. J Natl Cancer Inst. Jan 21; 2009 101(2):114–119. [PubMed: 19141773] 27. Schroder FH, Hugosson J, Roobol MJ, et al. Screening and prostate-cancer mortality in a randomized European study. N Engl J Med. Mar 26; 2009 360(13):1320–1328. [PubMed: 19297566] 28. Andriole GL, Crawford ED, Grubb RL 3rd, et al. Mortality results from a randomized prostatecancer screening trial. N Engl J Med. Mar 26; 2009 360(13):1310–1319. [PubMed: 19297565] 29. Vickers AJ. Decision analysis for the evaluation of diagnostic tests, prediction models and molecular markers. Am Stat. 2008; 62(4):314–320. [PubMed: 19132141] 30. Vickers AJ, Jang K, Sargent D, Lilja H, Kattan MW. Systematic review of statistical methods used in molecular marker studies in cancer. Cancer. Apr 15; 2008 112(8):1862–1868. [PubMed: 18320601] 31. Leman ES, Cannon GW, Trock BJ, et al. EPCA-2: a highly specific serum marker for prostate cancer. Urology. Apr; 2007 69(4):714–720. [PubMed: 17445657] 32. New Prostate Cancer Test May Detect More Tumors. The Washington Post. Apr.2007 26 33. Szabo L. Test shows promise in detecting prostate cancer. USA Today. Apr 26.2007 34. Zheng SL, Sun J, Wiklund F, et al. Cumulative association of five genetic variants with prostate cancer. N Engl J Med. Feb 28; 2008 358(9):910–919. [PubMed: 18199855] 35. Kolata G. $300 to Learn RIsk of Prostate Cancer. The New York Times. Jan 17.2008 36. Vickers A, Lilja H, Scardino P. Five genetic variants associated with prostate cancer. N Engl J Med. Jun 19.2008 358(25):2740. author reply 2741. [PubMed: 18572449] 37. Lilja H, Ulmert D, Bjork T, et al. Long-term prediction of prostate cancer up to 25 years before diagnosis of prostate cancer using prostate kallikreins measured at age 44 to 50 years. J Clin Oncol. Feb 1; 2007 25(4):431–436. [PubMed: 17264339] 38. Salinas CA, Koopmeiners JS, Kwon EM, et al. Clinical utility of five genetic variants for predicting prostate cancer risk and mortality. Prostate. Mar 1; 2009 69(4):363–372. [PubMed: 19058137] 39. Kattan MW, Wheeler TM, Scardino PT. Postoperative nomogram for disease recurrence after radical prostatectomy for prostate cancer. J Clin Oncol. May; 1999 17(5):1499–1507. [PubMed: 10334537] 40. Stephenson AJ, Scardino PT, Eastham JA, et al. Postoperative nomogram predicting the 10-year probability of prostate cancer recurrence after radical prostatectomy. J Clin Oncol. Oct 1; 2005 23(28):7005–7012. [PubMed: 16192588]
Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 11
NIH-PA Author Manuscript NIH-PA Author Manuscript
41. Eggener SE, Vickers AJ, Serio AM, et al. Comparison of models to predict clinical failure after radical prostatectomy. Cancer. Jan 15; 2009 115(2):303–310. [PubMed: 19025977] 42. Aureon Laboratories. Prognostic Tests. Prostate Px. http://aureon.com/prognostic-tests-prostatepx.htm 43. Carter HB, Ferrucci L, Kettermann A, et al. Detection of life-threatening prostate cancer with prostate-specific antigen velocity during a window of curability. J Natl Cancer Inst. Nov 1; 2006 98(21):1521–1527. [PubMed: 17077354] 44. Prostate cancer detection (Version 2.2007), National Comprehensive Cancer Network. 2007. (www.nccn.org) 45. Vickers AJ, Savage C, O'Brien MF, Lilja H. Systematic review of pretreatment prostate-specific antigen velocity and doubling time as predictors for prostate cancer. J Clin Oncol. Jan 20; 2009 27(3):398–403. [PubMed: 19064972] 46. Loeb S, Roehl KA, Catalona WJ, Nadler RB. Prostate specific antigen velocity threshold for predicting prostate cancer in young men. J Urol. Mar; 2007 177(3):899–902. [PubMed: 17296371] 47. Roddam AW, Duffy MJ, Hamdy FC, et al. Use of prostate-specific antigen (PSA) isoforms for the detection of prostate cancer in men with a PSA level of 2-10 ng/ml: systematic review and metaanalysis. Eur Urol. Sep; 2005 48(3):386–399. discussion 398-389. [PubMed: 15982797] 48. Becker C, Piironen T, Pettersson K, Hugosson J, Lilja H. Clinical value of human glandular kallikrein 2 and free and total prostate-specific antigen in serum from a population of men with prostate-specific antigen levels 3.0 ng/mL or greater. Urology. May; 2000 55(5):694–699. [PubMed: 10792083] 49. Pajak TF, Clark GM, Sargent DJ, McShane LM, Hammond ME. Statistical issues in tumor marker studies. Arch Pathol Lab Med. Jul; 2000 124(7):1011–1015. [PubMed: 10888777] 50. Cronin AM, Vickers AJ. Statistical methods to correct for verification bias in diagnostic studies are inadequate when there are few false negatives: a simulation study. BMC Med Res Methodol. 2008; 8:75. [PubMed: 19014457] 51. Etzioni R. Statistical issues in the evaluation of screening and early detection modalities. Urol Oncol. May-Jun;2008 26(3):308–315. [PubMed: 18452826] 52. McShane LM, Altman DG, Sauerbrei W, et al. Reporting recommendations for tumor marker prognostic studies (REMARK). J Natl Cancer Inst. Aug 17; 2005 97(16):1180–1184. [PubMed: 16106022]
NIH-PA Author Manuscript Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 12
Table 1
NIH-PA Author Manuscript
Stages of marker development 1.
Can the marker be measured accurately and reproducibly?
2.
Does the marker distinguish between convenience samples of clearly distinguishable groups of patients?
3.
Is the marker associated with outcome in the sort of patients to whom the marker would be applied to in practice?
4.
Does the marker provide additional information to that already available to the clinician?
5.
Does the use of the marker improve clinical outcome?
NIH-PA Author Manuscript NIH-PA Author Manuscript Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 13
Table 2
NIH-PA Author Manuscript
Two hypothetical studies that use convenience samples of subjects without disease, benign disease, localized, or advanced cancer The marker's sensitivity and specificity for advanced disease and those without disease is high (both 90%); however, the marker is unable to correctly patients with localized disease or those with a benign condition (sensitivity and specificity are 50%). A) Equal numbers of patients are included in each group (healthy, benign, local cancer and advanced cancer). The overall sensitivity and specificity are both 70%. B) Sampling of patients reflects the prevalence of groups seen in the clinic. In this example 25% of patients have no disease, 50% have a benign condition, 20% of patients have localized cancer and 5% have advanced cancer. The sensitivity is 58% and the specificity is 63%. A) Cancer (Advanced or Localized)
No Cancer (No disease or benign condition)
Marker +
90% × 500 + 50% × 500 = 700 (True positives)
10% × 500 + 50% × 500 = 300 (False positives)
Marker -
10% × 500 + 50% × 500 = 300 (False negatives)
90% × 500 + 50% × 500 = 700 (True negatives)
1000
1000
NIH-PA Author Manuscript
B)
Marker + Marker -
Cancer (Advanced or Localized)
No Cancer (No disease or benign condition)
90% × 50 + 50% × 200 = 145 (True positives)
10% × 250 + 50% × 500 = 275 (False positives)
10% × 50 + 50% × 200 = 105 (False negatives)
90% × 250 + 50% × 500 = 475 (True negatives)
250
750
NIH-PA Author Manuscript Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
Savage and Vickers
Page 14
Table 3
NIH-PA Author Manuscript
Hypothetical results of a marker study for prostate biopsy, showing a decision analysis. Strategy
Cancers found
Unnecessary Biopsies
Net benefit
Biopsy all men with elevated PSA
300
700
300 - 700 ÷ 4=125
Biopsy if risk ≥ 20% from statistical model based on molecular marker
210
300
210 - 300 ÷ 4=135
NIH-PA Author Manuscript NIH-PA Author Manuscript Grand rounds Urol. Author manuscript; available in PMC 2014 May 13.
