J ClinEpidemiol Vol. 43,No. 8,pp. 753-754,1990 Printed in Great Britain. All rights reserved
Copyright 0
0895-4356/90 $3.00+O.OO 1990 Pergamon Press plc
Variance and Dissent: Dissent A REVEAL-CONCEAL TEST FOR MANUSCRIPT REVIEW: ITS APPLICATION IN THE OBESITY AND MORTALITY STUDY ALFRED A. RIMM Division of Biostatistics/Clinical Epidemiology, Medical College of Wisconsin, Milwaukee, WI 53226, U.S.A. (Received for publication 23 January 1990)
In the study by Wilcosky et al. (J Clin Epidemiol 1990; 43: 743-752) the main hypothesis is that body mass index (BMI) is associated with the incidence of death or mortality rate. Though the key outcome variable in this study is mortality rate, no rates are given. As a referee of this paper, I requested that they be included. One of the leaders of the lipid study, a renowned epidemiologist (not a co-author) responded to the request by stating that the statisticians did not think that they were necessary and he said that he deferred to their judgment. My immediate reaction was that this was one of the best examples I have ever seen of the biostatistical tail wagging the epidemiologic dog. It is unfortunate to see the epidemiologist being converted to the acceptance of model deification, because this has led to the concealing of basic data. Apparently to this epidemiologist things looked scientific because there are three tables of regression coefficients and some of them were even carried out to four places to the right of the decimal. When you have been refereeing manuscripts for a couple of decades and you have a few hundred under your belt, as I do, you develop criteria for making decisions. I would like to discuss one of these criteria as it is relevant to the work of Wilcosky et al. The work passed all of my criteria but one, called the reveal-conceal test. This criterion is especially useful for secondary analyses such as presented in their 753
work. In these studies the authors come equipped with data, computers, software and models and then usually churn out yard thick outputs. The authors must then sort through all of the tables of results and then decide which ones to reveal to the reader. This is no easy task when you have to decide among the hundreds of tables that come out of the computer. In general these tables may be grouped into five categories: (1) redundant, (2) uninteresting or null, (3) irrelevant, (4) clear answer to a question (usually revealed to the readers) and (5) mavericks. The tables in the last group do not agree with the main thrust of the key findings of the study and/or the well established findings in the literature and/or point up problems with the data. What do you do with the maverick tables? First of all, you wish that you had not run them. But since you did and then laid your eyes on them you have got to explain to yourself why you are not going to use them. The process of explaining away or concealing maverick tables has its foundations in religion. That is, you develop a set of beliefs based upon your teachings and experiences: then you let your beliefs justify your behavior. It is easy for a scientist to adopt a firm belief in a particular model or process that is purported to explain a complex biomedical system. Unfortunately, some strong beliefs do not have solid factual bases, and in spite of this may influence the behavior of good scientists. Thus, it is possible that a “believing”
754
ALFREDA.RIMM
Tables
Findings
support
behavior
some
Tables
of
model
support
Reveal
Conceal
I
I
I
on
strongly model
I
I I
Revea 1
request
(
Referee’s decision
give
support
model
I \ Authors’
Tables
do not
Fail
review
k-
Pass
review
(
Fig. 1. Theoretical distribution of tables of results and author’s behavior along with referee’s decision point.
scientist may justify the discarding of maverick tables. To put this in an overall perspective, Fig. 1 overleaf shows the scaling of tables according to their degree of support of a hypothesis or model and the behavior or action taken by the scientist. It also shows the referee’s action based upon his/ her request for additional data from authors. This model fits the present situation well. When I requested the mortality rates from the authors they refused to reveal them and I suggested that the paper be rejected. I made the assumption that the data did not support their model: I had no other choice. I presume that we are expected to view their work as good science because the models are being offered as an explanation of phenomena in the real world. It is difficult to do this because the authors admit that the quadratic curves do not really explain the data very well because there is a sharp rise in mortality at the front end of the BMI scale. Furthermore, they point out that the actual data are asymmetric even though
the models they are using do not deal with this problem. In my opinion the models in this study arise more from beliefs than from rigorous scientific evidence, but without the hard data it is difficult to know. So, the models and their 58 regression coefficients in their Tables 5, 6 and 7 and the curves in their Fig. 1 are what the authors were willing to reveal. There is a suspicion but no proof that the concealed mortality rates would affect our opinion of their results. The fact that they are not revealed suggests that they would not, at least from a visual point of view, support their key findings. In these situations, I would like to propose that authors share their data with referees for further analyses. There are court cases in which the judge has asked the plaintiffs statistician, defense’s statistician and the court’s statistician to analyze the same body of data to resolve a problem. I would like to see this implemented by the Journal in special situations. I believe this proposal would expedite our search for the truth.
Copyright 0
0895-4356/90 $3.00+O.OO 1990 Pergamon Press plc
Variance and Dissent: Dissent A REVEAL-CONCEAL TEST FOR MANUSCRIPT REVIEW: ITS APPLICATION IN THE OBESITY AND MORTALITY STUDY ALFRED A. RIMM Division of Biostatistics/Clinical Epidemiology, Medical College of Wisconsin, Milwaukee, WI 53226, U.S.A. (Received for publication 23 January 1990)
In the study by Wilcosky et al. (J Clin Epidemiol 1990; 43: 743-752) the main hypothesis is that body mass index (BMI) is associated with the incidence of death or mortality rate. Though the key outcome variable in this study is mortality rate, no rates are given. As a referee of this paper, I requested that they be included. One of the leaders of the lipid study, a renowned epidemiologist (not a co-author) responded to the request by stating that the statisticians did not think that they were necessary and he said that he deferred to their judgment. My immediate reaction was that this was one of the best examples I have ever seen of the biostatistical tail wagging the epidemiologic dog. It is unfortunate to see the epidemiologist being converted to the acceptance of model deification, because this has led to the concealing of basic data. Apparently to this epidemiologist things looked scientific because there are three tables of regression coefficients and some of them were even carried out to four places to the right of the decimal. When you have been refereeing manuscripts for a couple of decades and you have a few hundred under your belt, as I do, you develop criteria for making decisions. I would like to discuss one of these criteria as it is relevant to the work of Wilcosky et al. The work passed all of my criteria but one, called the reveal-conceal test. This criterion is especially useful for secondary analyses such as presented in their 753
work. In these studies the authors come equipped with data, computers, software and models and then usually churn out yard thick outputs. The authors must then sort through all of the tables of results and then decide which ones to reveal to the reader. This is no easy task when you have to decide among the hundreds of tables that come out of the computer. In general these tables may be grouped into five categories: (1) redundant, (2) uninteresting or null, (3) irrelevant, (4) clear answer to a question (usually revealed to the readers) and (5) mavericks. The tables in the last group do not agree with the main thrust of the key findings of the study and/or the well established findings in the literature and/or point up problems with the data. What do you do with the maverick tables? First of all, you wish that you had not run them. But since you did and then laid your eyes on them you have got to explain to yourself why you are not going to use them. The process of explaining away or concealing maverick tables has its foundations in religion. That is, you develop a set of beliefs based upon your teachings and experiences: then you let your beliefs justify your behavior. It is easy for a scientist to adopt a firm belief in a particular model or process that is purported to explain a complex biomedical system. Unfortunately, some strong beliefs do not have solid factual bases, and in spite of this may influence the behavior of good scientists. Thus, it is possible that a “believing”
754
ALFREDA.RIMM
Tables
Findings
support
behavior
some
Tables
of
model
support
Reveal
Conceal
I
I
I
on
strongly model
I
I I
Revea 1
request
(
Referee’s decision
give
support
model
I \ Authors’
Tables
do not
Fail
review
k-
Pass
review
(
Fig. 1. Theoretical distribution of tables of results and author’s behavior along with referee’s decision point.
scientist may justify the discarding of maverick tables. To put this in an overall perspective, Fig. 1 overleaf shows the scaling of tables according to their degree of support of a hypothesis or model and the behavior or action taken by the scientist. It also shows the referee’s action based upon his/ her request for additional data from authors. This model fits the present situation well. When I requested the mortality rates from the authors they refused to reveal them and I suggested that the paper be rejected. I made the assumption that the data did not support their model: I had no other choice. I presume that we are expected to view their work as good science because the models are being offered as an explanation of phenomena in the real world. It is difficult to do this because the authors admit that the quadratic curves do not really explain the data very well because there is a sharp rise in mortality at the front end of the BMI scale. Furthermore, they point out that the actual data are asymmetric even though
the models they are using do not deal with this problem. In my opinion the models in this study arise more from beliefs than from rigorous scientific evidence, but without the hard data it is difficult to know. So, the models and their 58 regression coefficients in their Tables 5, 6 and 7 and the curves in their Fig. 1 are what the authors were willing to reveal. There is a suspicion but no proof that the concealed mortality rates would affect our opinion of their results. The fact that they are not revealed suggests that they would not, at least from a visual point of view, support their key findings. In these situations, I would like to propose that authors share their data with referees for further analyses. There are court cases in which the judge has asked the plaintiffs statistician, defense’s statistician and the court’s statistician to analyze the same body of data to resolve a problem. I would like to see this implemented by the Journal in special situations. I believe this proposal would expedite our search for the truth.
