Evaluating a Training Institute for Juvenile Court Judges Gilbert Geis, Ph.D. Charles W. Tenney, Jr., LL.M.
ABSTRACT: Increasing attention of late has been directed toward determining the dynamics and the effectiveness of training institutes. Such evaluations provide an indication of impact of specified kinds of material and of outcomes sometimes unanticipated by the conference planners. They allow for refashioning of conference content and for comparative statements regarding subsequent endeavors. The present paper reviews evaluative techniques employed to monitor a training course offered to juvenile court judges, assessing their utility and indicating some of the findings. Logbooks, questionnaires, inquiries submitted by conference speakers, and similar tools were used. The authors, who did not take part in the program but served as evaluators, suggest that, taken together, the various evaluative approaches appear to offer a rounded portrait of the events and consequences of the training institute under review and an approach that might be useful for similar undertakings.
Juvenile courts, charged with the welfare of children, represent in many respects, at one and the same time, the most promising and the most dangerous judicial development in the past half-century. They stand as an experiment whose aim is to determine whether the insights and the knowledge of the behavioral sciences can combine with the traditional protections inherent in the legal order to form an amalgam doing injustice to neither ingredient and resulting in enhanced assistance to the children of the society and, ultimately, through them, to all members of the society. It has been a long-term experiment, measured in years and in the lives of persons caught up in the process, though no certain answer to the issues the juvenile court presents is yet at hand (Sheridan, ~966; Rosenheim, x962; Antieau, x96~). Ultimately, all lines of the operation of the juvenile court lead to the judge or emanate from him. As Killian (i949) had indicated: "No court can rise above its judge--particularly a separate and independent court." The same writer stresses that juvenile court judges come to their job burdened with innumerable handicaps, induding those involving tenure, salary, method of selection, status, and, most notably, preparation for the job they will be called upon to perform. "Juvenile court judges," Killian (~949, P. 95) points out, "more Dr. Gels is Professor of Sociology, California State College, Los Angeles, and Mr. Tenney is Associate Professor, University of Nebraska Law College, Lincoln, Nebraska. Community Mental Health Journal, Vol. 4 (6), 1968
461
462
Community Mental Health Journal
than others m u s t . . , acquire judgment with respect to the exacting application of legal and social norms, and experience in a dawning sense of the law in relation to the social disciplines." These judicial handicaps are even more evident in rural jurisdictions where, according to one acerbic critic, "nine out of ten of our juvenile courts continue so far below grade as to be practically subterranean" (Carr, :~949, P. 3z) 9 Considerations such as the foregoing prompted the National Council of Juvenile Court Judges to stage a two-week training session for 3 ~ juvenile court judges from predominantly rural areas, about 4~ percent of whom had been on the bench for less than one year. The primary purpose of the conference, held at the University of Colorado Law School during the summer of 5965 , was to acquaint these rural juvenile court judges with current ideas and information prevalent in sociology, psychiatry, and social work regarding human behavior, and particularly that behavior that appeared to lie behind juvenile delinquency. Attention was also devoted at the institute to a combination of lectures and exercises in sensitivity training (Bradford, 5964; Sacks, 5959) and to a number of debates among the participants concerning various legal issues involved in the operation of a juvenile court. The need to determine the outcome of the conference with more exactitude than is normally involved in the assessment of such endeavors is what underlay the multifaceted evaluation of the training institute. The techniques employed, since they appear to have general relevance for evaluative purposes, and the results returned, which bear upon the efficacy of the evaluative instruments, constitute the basis of the present report. As Hyman and his collaborators (~962, p. 5) have pointed out: "Ours is an age of action programs" and for management of such enterprises, evaluation represents "the proper methodological accompaniment to rational action." The evaluation approach employed during the institute for rural juvenile court judges unearthed considerable information regarding reactions to the institute itself as well as a good deal of data bearing more directly on communicative processes, and particularly the relationships among the kinds of persons in attendance, the content of the material presented to them, and their responses to such material. The evaluative tools employed can be grouped under four headings: (5) logbooks maintained by the conference participants; (z) evaluative inquiries submitted beforehand by the conference speakers; (3 & 4) pre- and post-conference questionnaires submitted to the participants requesting biographical data, expectations regarding the conference, patterns of work, and opinions concerning diverse aspects and problems involved in being a juvenile court judge. This questionnaire also contained three truncated attitude inventories regarding (a) ethnocentrism; (b) anomie; and (c) views on various issues in law enforcement and the administration of justice. LOGBOOKS KEPT BY PARTICIPANTS Each of the 30 judges was provided with a notebook and a good deal of urging and constant reminding to record in it (as the conference
Gilbert Gels and Charles W. Tenney, Jr.
463
unfolded and at various times when he had the leisure) his reactions to each program element. The logbooks, it was suggested, could contain summaries of the speakers' views, points with which the writer differed, suggestions regarding amendments to the conference schedule, and summary judgments concerning those things that the judge was seeing and hearing. For evaluative purposes the logbooks provided a lode of material that never appeared either in formal discussions or in informal conversations. Several of the judges, for example, wrote that they were offended by the blunt sexual content of the psychiatric presentation, though during the sessions themselves they carefully camouflaged this personal malaise. The logbooks also dearly indicated that many items that the staff and the speakers were stressing were not necessarily the same points catching the judges' fancy. Arguments with the speakers, sharp remarks at some of their personal idiosyncracies, and a plethora of judgmental material pervaded the logbooks, and provided a rich perspective for analysis of the program content that otherwise would have been impossible to achieve. In addition, a good deal of ongoing comment as each speaker proceeded through his segment of the program was highly illuminating concerning changes in reactions over time. Initial skepticism toward the sensitivity training among the judges--they felt that they were being forced to play children's games and were being manipulated (the term "guinea pig" showed up almost routinely in the logbooks)--changed gradually to a final feeling of relevance and acceptance. Stinging hostility toward the faculty sociologist during his initial talks was later transformed to grudging respect and outright admiration among the judges. From the logbooks it was possible to discern those ingredients, even among the first horrified reactions, that would allow room for eventual acceptance, if the sociologist could tap with some grace the latent sources of approval. In the case of the sociological presentation, for instance, the following reactive pattern could fairly readily be discovered. The biographical data derived from the judges had shown their almost total lack of training in social science, and their underlying antipathy ("Which college courses did you like least?" they had been asked) to humanistic and natural science material. The anomie scale had shown, in addition, that the judges felt extremely closely integrated into their home communities. Into such a setting, the sociologist came forward with direct vehemence (as the logbooks reported vividly) in his insistence that the judges really did not understand their home towns, particularly the impoverished and deprived segments of these towns' populations. At the same time, the logbooks showed a particular tendency among the judges toward acceptance of polemic views and argumentation, a characteristic almost certainly a product of their legal training and legal careers (Riesman, "r962 ) so that their reservations about what the sociologist was saying were tinged with doubt as if uncertain how a jury might react, and an admiration for the verve and the unrelenting polemics of the presentation. Many other evaluative advantages garnered from the logbooks are worth
464
Community Mental Health Journal
brief mention. They often added humor to the evaluation, as judges took time to scribble down amused reactions to different events. They brought forward material well beyond the ken of formal tools of analysis. They probably also served somewhat to give a feeling of importance to the participants and to place greater emphasis on their views. It is also possible, since it had been announced that they would be collected, that the logbooks increased attentiveness to the program among the participants so that they could make a presentable logbook showing. QUESTIONS SUBMITTED BY SPEAKERS It is, after all, the speaker himself who should know best what he intends to convey to his audience and what attitudes he presumes will ensue from such a presentation. Each speaker therefore was requested to submit questions to be answered by conference participants. Such questions were put to the judges prior to the opening of the training sessions and then again several weeks afterwards. The speakers' questions proved to be of considerable evaluative utility. Perhaps the major conclusion reached was that the speakers more often than not failed to address those items that they had earlier indicated (in terms of their questions) that they would stress. On many occasions as well, though they emphasized points included in their questions, they failed to appreciate, as the first inquiries had disclosed to the evaluators, that there was little disagreement between them and the judges on certain matters. The social worker, for instance, had asked the judges to list the advantages of trained social workers over the untrained variety. The answers were thorough and articulate. Unaware of this, the speaker proceeded to outline with force and spirit the importance of trained workers, and argued strongly that they were well worth the price differential that they would command. Following the conference, the judges again presented essentially the same views concerning the superiority of trained social workers, though these had been tightened up somewhat. But several participants now entered rather querulous notes that such workers really were "overtrained" and "inappropriate" for the kind of court operations that they conducted, and the way in which they worded their responses left little doubt that they had been intimidated and had abandoned their previous accepting position through the very fervor of the social work presentation. It was, of course, much easier to tabulate responses that were in the form of checklists rather than being open-ended. The sensitivity-training speaker had submitted such a list, and five of the six items on it showed movement in the direction toward which he had been aiming. The dearest "improvement" was an almost doubled (on a minus 3 to plus 3 scale) agreement with the point: "Since the individual is ultimately responsible for a decision, the decision must be his and his alone." Brief discussion questions were, of course, more difficult to analyze, though it was impressive to note how, with the lapse of several months, the respondents repeated almost verbatim long and elaborate replies that had appeared in their original responses.
Gilbert Geis and Charles W. Tenney, Jr.
465
The speakers' questions would seem to provide them with a useful indication of the impact that they are able to produce in terms of those items that they believe are important. Perhaps in the future it will be found more useful to submit to potential speakers the first round of preconference replies, and challenge them to see if they can, within certain ground rules, produce a more "favorable" response. Used either way, the approach seems to offer a good deal of evaluative assistance. PRE- AND POSTCONFERENCE QUESTIONNAIRES The utility of examining convictions and attitudes held before the training sessions and those present afterwards hardly needs elaboration. The belief, widely held, that little of measurable value could be ascertained following so brief an exposure to program content proved to be inaccurate in regard to the two-week training period for rural juvenile court judges, though there is no gainsaying that it is a long and most important step to determine whether registered alterations in questionnaire responses represent actual changes in relevant behavior (Deutscher, -r966 ). Severe budget limitations precluded any possibility of a more sophisticated approach that would have permitted monitoring the judges' performances in their courtroom prior to and following the training institute in order to obtain a behavioral portrait of the conference impact. The pre- and postconference questionnaires found that the initial belief that the social work and psychiatric presentations would be most useful for the court work (sensitivity training was not included in this inquiry because of the judges' presumed unfamiliarity with it prior to the conference) had changed to a postconference conviction, by a large margin, that the sociological material had contained the "most meaning" for courtroom work. Obviously, the blend of the speaker himself and the material he presented undercuts any possibility of maintaining that it was the subject matter alone--either as presented, or more remotely, as contained in the discipline--that had converted the judges. In the face of the judges' initial antagonism to the sociological presentation, and their only grudging acceptance of it by the time it concluded, the striking subsequent endorsement invited evaluative speculation. Probably there was a retroactive glow afforded the material, which could have been a testament to the speaker's unquestioned ability to overcome that most deadly of all conference items--boredom. Probably the increased distance between the judges and the irritating and threatening remarks of the sociologist made them more acceptable, and possibly the views had taken on added value by being tested and found important when the judges returned to their home jurisdictions. The various kinds of material collected in the 'evaluation, though they might not provide definitive answers to issues such as these, in most instances allowed for reasonably informed speculation concerning them. The postconference questionnaires also showed that the judges had moved somewhat toward a more conservative view of the possible impact of the conference on their courtroom performance. Beforehand, a majority had thought
4SS
Community Mental Health Journal
the conference would effect their court operation "a good deal." Now most believed it had effected the way they ran their court "quite a bit," a lesser endorsement. The report of impact was particularly strong among the newer judges, those with less than a year of service. On the other hand, detailed questions concerning factors stressed as important in running a juvenile court showed clearly that it was the experienced judges who demonstrated the most alteration among such items, and primarily in regard to the "formality" they now imposed upon their courtroom hearings. The legal presentation at the conference had stressed the necessity to have greater regard for due process requirements in juvenile court. This lesson, the questionnaires discovered, had carried home to many experienced judges who, over the years, had become relaxed about such matters. The newer judges, more insecure, had been running their courts, at least according to their responses, more like the regular criminal and civil courts with which they had had earlier experience. The authoritarian scale showed a slight increase in that quality, produced in large measure by changes in responses to selected items, while the anomie scale indicated an increase of almost one-third above the original position of the judges, with increases quite uniform among the iz items on the scale submitted to the judges. These responses had for the conference sponsors a certain irony, though one which, on reflection, was not inhospitable to their aims. Speaker after speaker during the conference had entered a disclaimer for the ability of his field to solve the pressing dispositional dilemmas confronting the judges. Only the sensitivity trainer had taken a positive position, maintaining that more effective communication could assist the judges in doing a better job, but his message was often discounted because many of the judges believed that he was not taking adequate account of the statutory burden imposed upon them for unilateral decision-making. In terms of the presentations, the judges, as reflected in the anomie scale, obviously felt a good deal more isolated, dependent upon their own resources, and rather lonesome about it. Their checklist of "needs" of the court showed a striking change toward the view that they really could not depend upon diagnostic centers and psychological testing as much as they had earlier believed. While they still felt that they could use such ancillary information, they appeared much more willing, however reluctantly, to set out on their own. In a more positive vein, it was obvious that the conference had provided the judges with considerable support and reassurance that, while they may not be altogether certain about the validity of what they were doing, second-guessers in the behavioral sciences shared this vocational inexactness with them. It is, of course, a very delicate and most important issue to determine when ignorance becomes disabling and when humility becomes enabling. But in terms of the historical development of the juvenile court, the increased selfassurance of the judges and the weaning from blind subservience to behavioral science judgments may be viewed as a tentative step in the right direction, however much longer and intricate the path toward the most desirable resolution may be. For, as Francis Allen (~958) has pointed out, a major allegation
Gilbert Geis and Charles W. Tenney, Jr.
467
against juvenile courts is that they tend to interfere with the liberty of persons on the basis of diagnostic information and insights that overextend the competence of the behavioral sciences putting them forward. EVALUATIVE PANEL Another evaluative procedure that seemed to be both productive and rewarding involved a formal summary session at the end of the conference during which a panel of judges put before the total group the considered judgments of themselves and their fellows regarding a number of specific questions that had been addressed to them. The judges were divided into three groups that privately prepared summary statements regarding their consensus on various questions. While the technique undoubtedly blunted the more extreme views, it served to force the participants to examine together at some length specific conference items. Many of the panel's remarks came to grips with technical aspects of the conference, including things such as the schedule for meals, mileage allowances, and similar items. The participants indicated that they had wanted maps of the area provided beforehand, and made many other suggestions of considerable value to conference planners contemplating subsequent institutes. There were also summary statements on the speakers and their material, most of which echoed logbook comments. The evaluation panel, providing a formal tie-up of the conference, seemed to lend an air of closure and neatness and well-being to the package, besides permitting the group to be reasonably gracious, now that it was over, in their expressions of approval of those things they had particularly liked. The camaraderie that it brought about was probably equally as important as the evaluative material that it produced. CONCLUSIONS The general aim of the evaluation of the two-week training institute for rural juvenile court judges was to provide a running commentary on the program so that a feeling for it and for the responses of the judges to it could be gained. There was no attempt to draw invidious distinctions between various elements of the program, except in terms of suitability for future conferences or usefulness to individuals interested in general principles of communication or the management of training programs. The techniques employed for evaluation seemed to indicate with some clarity that it is possible, using relatively simple and inexpensive methods, to gain a measure of understanding of responses to program elements and the possible effect that a brief training institute produces upon a given audience. The logbooks provided facts and views of descriptive value, though it was always necessary to guard against seduction by colorful yet idiosyncratic views. The questions submitted beforehand by the various speakers would seem to present an evaluative approach of considerable merit, not only for tapping audience reaction but also for determining whether a speaker addresses himself to those issues he maintains he will talk to, and whether he is correct in assessing
468
Community Mental Health Journal
the position of his audience regarding the given issue and, finally, whether he is able to produce the kind of changes he is seeking. The pre- and postconference questionnaires seem to provide information that adds depth and understanding regarding the participants' more lasting reactions to the different kinds of material. For the juvenile court judges' institute, the postconference questionnaires showed that alterations can be expected beyond chance variation though, if resources were available, it would have been preferable to have measured alterations through the use of control groups and blind and double-blind experimental approaches. The employment of the different evaluative tools, as reported here, was motivated in large measure by the uncertainty regarding the greater efficacy or superior value of any one of them. The exercise perhaps demonstrated that no single approach can hope to provide a rounded portrait concerning even so brief a period of training as that given the juvenile court judges; taken together, they may begin to fill in some of the important gaps and to provide complementary and supplementary views of similar items. Obviously, too, more and much better evaluative approaches are needed, though it will remain something of an issue to determine whether the investment of time and money to develop and carry out an evaluation of high art and precise science is a reasonable venture for a transient training program. The aim here was exploratory, to see what could be done with limited resources, and what ideas could be generated under such conditions that would make the effort worthwhile, the results useful, and the techniques worth recommendation. REFERENCES Allen, F. A. The borderland of the criminal law: Problems of "socializing" criminal law. Social Service Review, I958, 32, ~o7-127. Antiean, C. Constitutional rights in juvenile court. Cornell Law Quarterly, 2962, 46, 387-425. Bradford, L. P. et al. T-Group and laboratory method. New York: Wiley, 2964. Carr, L. J. Most courts have to be substandard. Federal Probation, ~949, I3, 29-33. Deutscher, I. Words and deeds: Social science and social policy. Social Problems, 2966, 23, 276-254. Hyman, H. et al. Applications of methods of evaluation. Berkeley: University of California Press, 2962. Killian, F. W. The juvenile court as an institution. The Annals, 2949, a62, 89-2oo. Riesman, D. Law and sociology: Recruitment, training, and colleagueship. In W. M. Evan, Law and sociology. New York: Free Press of Glencoe, 2962. Pp. z2-55. Rosenheim, M. K. (Ed.)Justice for the child. New York: FreePress of Glencoe, 2962. Sacks, H. R. Human relations training for lawyers and law students. Journal of Legal Education, 2959, 15, 726-736. Sheridan, W. H. Standards for juvenile and family courts. Children's Bureau Publication No. 437, ~966.
ABSTRACT: Increasing attention of late has been directed toward determining the dynamics and the effectiveness of training institutes. Such evaluations provide an indication of impact of specified kinds of material and of outcomes sometimes unanticipated by the conference planners. They allow for refashioning of conference content and for comparative statements regarding subsequent endeavors. The present paper reviews evaluative techniques employed to monitor a training course offered to juvenile court judges, assessing their utility and indicating some of the findings. Logbooks, questionnaires, inquiries submitted by conference speakers, and similar tools were used. The authors, who did not take part in the program but served as evaluators, suggest that, taken together, the various evaluative approaches appear to offer a rounded portrait of the events and consequences of the training institute under review and an approach that might be useful for similar undertakings.
Juvenile courts, charged with the welfare of children, represent in many respects, at one and the same time, the most promising and the most dangerous judicial development in the past half-century. They stand as an experiment whose aim is to determine whether the insights and the knowledge of the behavioral sciences can combine with the traditional protections inherent in the legal order to form an amalgam doing injustice to neither ingredient and resulting in enhanced assistance to the children of the society and, ultimately, through them, to all members of the society. It has been a long-term experiment, measured in years and in the lives of persons caught up in the process, though no certain answer to the issues the juvenile court presents is yet at hand (Sheridan, ~966; Rosenheim, x962; Antieau, x96~). Ultimately, all lines of the operation of the juvenile court lead to the judge or emanate from him. As Killian (i949) had indicated: "No court can rise above its judge--particularly a separate and independent court." The same writer stresses that juvenile court judges come to their job burdened with innumerable handicaps, induding those involving tenure, salary, method of selection, status, and, most notably, preparation for the job they will be called upon to perform. "Juvenile court judges," Killian (~949, P. 95) points out, "more Dr. Gels is Professor of Sociology, California State College, Los Angeles, and Mr. Tenney is Associate Professor, University of Nebraska Law College, Lincoln, Nebraska. Community Mental Health Journal, Vol. 4 (6), 1968
461
462
Community Mental Health Journal
than others m u s t . . , acquire judgment with respect to the exacting application of legal and social norms, and experience in a dawning sense of the law in relation to the social disciplines." These judicial handicaps are even more evident in rural jurisdictions where, according to one acerbic critic, "nine out of ten of our juvenile courts continue so far below grade as to be practically subterranean" (Carr, :~949, P. 3z) 9 Considerations such as the foregoing prompted the National Council of Juvenile Court Judges to stage a two-week training session for 3 ~ juvenile court judges from predominantly rural areas, about 4~ percent of whom had been on the bench for less than one year. The primary purpose of the conference, held at the University of Colorado Law School during the summer of 5965 , was to acquaint these rural juvenile court judges with current ideas and information prevalent in sociology, psychiatry, and social work regarding human behavior, and particularly that behavior that appeared to lie behind juvenile delinquency. Attention was also devoted at the institute to a combination of lectures and exercises in sensitivity training (Bradford, 5964; Sacks, 5959) and to a number of debates among the participants concerning various legal issues involved in the operation of a juvenile court. The need to determine the outcome of the conference with more exactitude than is normally involved in the assessment of such endeavors is what underlay the multifaceted evaluation of the training institute. The techniques employed, since they appear to have general relevance for evaluative purposes, and the results returned, which bear upon the efficacy of the evaluative instruments, constitute the basis of the present report. As Hyman and his collaborators (~962, p. 5) have pointed out: "Ours is an age of action programs" and for management of such enterprises, evaluation represents "the proper methodological accompaniment to rational action." The evaluation approach employed during the institute for rural juvenile court judges unearthed considerable information regarding reactions to the institute itself as well as a good deal of data bearing more directly on communicative processes, and particularly the relationships among the kinds of persons in attendance, the content of the material presented to them, and their responses to such material. The evaluative tools employed can be grouped under four headings: (5) logbooks maintained by the conference participants; (z) evaluative inquiries submitted beforehand by the conference speakers; (3 & 4) pre- and post-conference questionnaires submitted to the participants requesting biographical data, expectations regarding the conference, patterns of work, and opinions concerning diverse aspects and problems involved in being a juvenile court judge. This questionnaire also contained three truncated attitude inventories regarding (a) ethnocentrism; (b) anomie; and (c) views on various issues in law enforcement and the administration of justice. LOGBOOKS KEPT BY PARTICIPANTS Each of the 30 judges was provided with a notebook and a good deal of urging and constant reminding to record in it (as the conference
Gilbert Gels and Charles W. Tenney, Jr.
463
unfolded and at various times when he had the leisure) his reactions to each program element. The logbooks, it was suggested, could contain summaries of the speakers' views, points with which the writer differed, suggestions regarding amendments to the conference schedule, and summary judgments concerning those things that the judge was seeing and hearing. For evaluative purposes the logbooks provided a lode of material that never appeared either in formal discussions or in informal conversations. Several of the judges, for example, wrote that they were offended by the blunt sexual content of the psychiatric presentation, though during the sessions themselves they carefully camouflaged this personal malaise. The logbooks also dearly indicated that many items that the staff and the speakers were stressing were not necessarily the same points catching the judges' fancy. Arguments with the speakers, sharp remarks at some of their personal idiosyncracies, and a plethora of judgmental material pervaded the logbooks, and provided a rich perspective for analysis of the program content that otherwise would have been impossible to achieve. In addition, a good deal of ongoing comment as each speaker proceeded through his segment of the program was highly illuminating concerning changes in reactions over time. Initial skepticism toward the sensitivity training among the judges--they felt that they were being forced to play children's games and were being manipulated (the term "guinea pig" showed up almost routinely in the logbooks)--changed gradually to a final feeling of relevance and acceptance. Stinging hostility toward the faculty sociologist during his initial talks was later transformed to grudging respect and outright admiration among the judges. From the logbooks it was possible to discern those ingredients, even among the first horrified reactions, that would allow room for eventual acceptance, if the sociologist could tap with some grace the latent sources of approval. In the case of the sociological presentation, for instance, the following reactive pattern could fairly readily be discovered. The biographical data derived from the judges had shown their almost total lack of training in social science, and their underlying antipathy ("Which college courses did you like least?" they had been asked) to humanistic and natural science material. The anomie scale had shown, in addition, that the judges felt extremely closely integrated into their home communities. Into such a setting, the sociologist came forward with direct vehemence (as the logbooks reported vividly) in his insistence that the judges really did not understand their home towns, particularly the impoverished and deprived segments of these towns' populations. At the same time, the logbooks showed a particular tendency among the judges toward acceptance of polemic views and argumentation, a characteristic almost certainly a product of their legal training and legal careers (Riesman, "r962 ) so that their reservations about what the sociologist was saying were tinged with doubt as if uncertain how a jury might react, and an admiration for the verve and the unrelenting polemics of the presentation. Many other evaluative advantages garnered from the logbooks are worth
464
Community Mental Health Journal
brief mention. They often added humor to the evaluation, as judges took time to scribble down amused reactions to different events. They brought forward material well beyond the ken of formal tools of analysis. They probably also served somewhat to give a feeling of importance to the participants and to place greater emphasis on their views. It is also possible, since it had been announced that they would be collected, that the logbooks increased attentiveness to the program among the participants so that they could make a presentable logbook showing. QUESTIONS SUBMITTED BY SPEAKERS It is, after all, the speaker himself who should know best what he intends to convey to his audience and what attitudes he presumes will ensue from such a presentation. Each speaker therefore was requested to submit questions to be answered by conference participants. Such questions were put to the judges prior to the opening of the training sessions and then again several weeks afterwards. The speakers' questions proved to be of considerable evaluative utility. Perhaps the major conclusion reached was that the speakers more often than not failed to address those items that they had earlier indicated (in terms of their questions) that they would stress. On many occasions as well, though they emphasized points included in their questions, they failed to appreciate, as the first inquiries had disclosed to the evaluators, that there was little disagreement between them and the judges on certain matters. The social worker, for instance, had asked the judges to list the advantages of trained social workers over the untrained variety. The answers were thorough and articulate. Unaware of this, the speaker proceeded to outline with force and spirit the importance of trained workers, and argued strongly that they were well worth the price differential that they would command. Following the conference, the judges again presented essentially the same views concerning the superiority of trained social workers, though these had been tightened up somewhat. But several participants now entered rather querulous notes that such workers really were "overtrained" and "inappropriate" for the kind of court operations that they conducted, and the way in which they worded their responses left little doubt that they had been intimidated and had abandoned their previous accepting position through the very fervor of the social work presentation. It was, of course, much easier to tabulate responses that were in the form of checklists rather than being open-ended. The sensitivity-training speaker had submitted such a list, and five of the six items on it showed movement in the direction toward which he had been aiming. The dearest "improvement" was an almost doubled (on a minus 3 to plus 3 scale) agreement with the point: "Since the individual is ultimately responsible for a decision, the decision must be his and his alone." Brief discussion questions were, of course, more difficult to analyze, though it was impressive to note how, with the lapse of several months, the respondents repeated almost verbatim long and elaborate replies that had appeared in their original responses.
Gilbert Geis and Charles W. Tenney, Jr.
465
The speakers' questions would seem to provide them with a useful indication of the impact that they are able to produce in terms of those items that they believe are important. Perhaps in the future it will be found more useful to submit to potential speakers the first round of preconference replies, and challenge them to see if they can, within certain ground rules, produce a more "favorable" response. Used either way, the approach seems to offer a good deal of evaluative assistance. PRE- AND POSTCONFERENCE QUESTIONNAIRES The utility of examining convictions and attitudes held before the training sessions and those present afterwards hardly needs elaboration. The belief, widely held, that little of measurable value could be ascertained following so brief an exposure to program content proved to be inaccurate in regard to the two-week training period for rural juvenile court judges, though there is no gainsaying that it is a long and most important step to determine whether registered alterations in questionnaire responses represent actual changes in relevant behavior (Deutscher, -r966 ). Severe budget limitations precluded any possibility of a more sophisticated approach that would have permitted monitoring the judges' performances in their courtroom prior to and following the training institute in order to obtain a behavioral portrait of the conference impact. The pre- and postconference questionnaires found that the initial belief that the social work and psychiatric presentations would be most useful for the court work (sensitivity training was not included in this inquiry because of the judges' presumed unfamiliarity with it prior to the conference) had changed to a postconference conviction, by a large margin, that the sociological material had contained the "most meaning" for courtroom work. Obviously, the blend of the speaker himself and the material he presented undercuts any possibility of maintaining that it was the subject matter alone--either as presented, or more remotely, as contained in the discipline--that had converted the judges. In the face of the judges' initial antagonism to the sociological presentation, and their only grudging acceptance of it by the time it concluded, the striking subsequent endorsement invited evaluative speculation. Probably there was a retroactive glow afforded the material, which could have been a testament to the speaker's unquestioned ability to overcome that most deadly of all conference items--boredom. Probably the increased distance between the judges and the irritating and threatening remarks of the sociologist made them more acceptable, and possibly the views had taken on added value by being tested and found important when the judges returned to their home jurisdictions. The various kinds of material collected in the 'evaluation, though they might not provide definitive answers to issues such as these, in most instances allowed for reasonably informed speculation concerning them. The postconference questionnaires also showed that the judges had moved somewhat toward a more conservative view of the possible impact of the conference on their courtroom performance. Beforehand, a majority had thought
4SS
Community Mental Health Journal
the conference would effect their court operation "a good deal." Now most believed it had effected the way they ran their court "quite a bit," a lesser endorsement. The report of impact was particularly strong among the newer judges, those with less than a year of service. On the other hand, detailed questions concerning factors stressed as important in running a juvenile court showed clearly that it was the experienced judges who demonstrated the most alteration among such items, and primarily in regard to the "formality" they now imposed upon their courtroom hearings. The legal presentation at the conference had stressed the necessity to have greater regard for due process requirements in juvenile court. This lesson, the questionnaires discovered, had carried home to many experienced judges who, over the years, had become relaxed about such matters. The newer judges, more insecure, had been running their courts, at least according to their responses, more like the regular criminal and civil courts with which they had had earlier experience. The authoritarian scale showed a slight increase in that quality, produced in large measure by changes in responses to selected items, while the anomie scale indicated an increase of almost one-third above the original position of the judges, with increases quite uniform among the iz items on the scale submitted to the judges. These responses had for the conference sponsors a certain irony, though one which, on reflection, was not inhospitable to their aims. Speaker after speaker during the conference had entered a disclaimer for the ability of his field to solve the pressing dispositional dilemmas confronting the judges. Only the sensitivity trainer had taken a positive position, maintaining that more effective communication could assist the judges in doing a better job, but his message was often discounted because many of the judges believed that he was not taking adequate account of the statutory burden imposed upon them for unilateral decision-making. In terms of the presentations, the judges, as reflected in the anomie scale, obviously felt a good deal more isolated, dependent upon their own resources, and rather lonesome about it. Their checklist of "needs" of the court showed a striking change toward the view that they really could not depend upon diagnostic centers and psychological testing as much as they had earlier believed. While they still felt that they could use such ancillary information, they appeared much more willing, however reluctantly, to set out on their own. In a more positive vein, it was obvious that the conference had provided the judges with considerable support and reassurance that, while they may not be altogether certain about the validity of what they were doing, second-guessers in the behavioral sciences shared this vocational inexactness with them. It is, of course, a very delicate and most important issue to determine when ignorance becomes disabling and when humility becomes enabling. But in terms of the historical development of the juvenile court, the increased selfassurance of the judges and the weaning from blind subservience to behavioral science judgments may be viewed as a tentative step in the right direction, however much longer and intricate the path toward the most desirable resolution may be. For, as Francis Allen (~958) has pointed out, a major allegation
Gilbert Geis and Charles W. Tenney, Jr.
467
against juvenile courts is that they tend to interfere with the liberty of persons on the basis of diagnostic information and insights that overextend the competence of the behavioral sciences putting them forward. EVALUATIVE PANEL Another evaluative procedure that seemed to be both productive and rewarding involved a formal summary session at the end of the conference during which a panel of judges put before the total group the considered judgments of themselves and their fellows regarding a number of specific questions that had been addressed to them. The judges were divided into three groups that privately prepared summary statements regarding their consensus on various questions. While the technique undoubtedly blunted the more extreme views, it served to force the participants to examine together at some length specific conference items. Many of the panel's remarks came to grips with technical aspects of the conference, including things such as the schedule for meals, mileage allowances, and similar items. The participants indicated that they had wanted maps of the area provided beforehand, and made many other suggestions of considerable value to conference planners contemplating subsequent institutes. There were also summary statements on the speakers and their material, most of which echoed logbook comments. The evaluation panel, providing a formal tie-up of the conference, seemed to lend an air of closure and neatness and well-being to the package, besides permitting the group to be reasonably gracious, now that it was over, in their expressions of approval of those things they had particularly liked. The camaraderie that it brought about was probably equally as important as the evaluative material that it produced. CONCLUSIONS The general aim of the evaluation of the two-week training institute for rural juvenile court judges was to provide a running commentary on the program so that a feeling for it and for the responses of the judges to it could be gained. There was no attempt to draw invidious distinctions between various elements of the program, except in terms of suitability for future conferences or usefulness to individuals interested in general principles of communication or the management of training programs. The techniques employed for evaluation seemed to indicate with some clarity that it is possible, using relatively simple and inexpensive methods, to gain a measure of understanding of responses to program elements and the possible effect that a brief training institute produces upon a given audience. The logbooks provided facts and views of descriptive value, though it was always necessary to guard against seduction by colorful yet idiosyncratic views. The questions submitted beforehand by the various speakers would seem to present an evaluative approach of considerable merit, not only for tapping audience reaction but also for determining whether a speaker addresses himself to those issues he maintains he will talk to, and whether he is correct in assessing
468
Community Mental Health Journal
the position of his audience regarding the given issue and, finally, whether he is able to produce the kind of changes he is seeking. The pre- and postconference questionnaires seem to provide information that adds depth and understanding regarding the participants' more lasting reactions to the different kinds of material. For the juvenile court judges' institute, the postconference questionnaires showed that alterations can be expected beyond chance variation though, if resources were available, it would have been preferable to have measured alterations through the use of control groups and blind and double-blind experimental approaches. The employment of the different evaluative tools, as reported here, was motivated in large measure by the uncertainty regarding the greater efficacy or superior value of any one of them. The exercise perhaps demonstrated that no single approach can hope to provide a rounded portrait concerning even so brief a period of training as that given the juvenile court judges; taken together, they may begin to fill in some of the important gaps and to provide complementary and supplementary views of similar items. Obviously, too, more and much better evaluative approaches are needed, though it will remain something of an issue to determine whether the investment of time and money to develop and carry out an evaluation of high art and precise science is a reasonable venture for a transient training program. The aim here was exploratory, to see what could be done with limited resources, and what ideas could be generated under such conditions that would make the effort worthwhile, the results useful, and the techniques worth recommendation. REFERENCES Allen, F. A. The borderland of the criminal law: Problems of "socializing" criminal law. Social Service Review, I958, 32, ~o7-127. Antiean, C. Constitutional rights in juvenile court. Cornell Law Quarterly, 2962, 46, 387-425. Bradford, L. P. et al. T-Group and laboratory method. New York: Wiley, 2964. Carr, L. J. Most courts have to be substandard. Federal Probation, ~949, I3, 29-33. Deutscher, I. Words and deeds: Social science and social policy. Social Problems, 2966, 23, 276-254. Hyman, H. et al. Applications of methods of evaluation. Berkeley: University of California Press, 2962. Killian, F. W. The juvenile court as an institution. The Annals, 2949, a62, 89-2oo. Riesman, D. Law and sociology: Recruitment, training, and colleagueship. In W. M. Evan, Law and sociology. New York: Free Press of Glencoe, 2962. Pp. z2-55. Rosenheim, M. K. (Ed.)Justice for the child. New York: FreePress of Glencoe, 2962. Sacks, H. R. Human relations training for lawyers and law students. Journal of Legal Education, 2959, 15, 726-736. Sheridan, W. H. Standards for juvenile and family courts. Children's Bureau Publication No. 437, ~966.
