Journal of Primary Prevention, 2(4), Summer, 1982
The Evaluation of a Kindergarten Social Problem Solving Program JANICE I. WINER, PAMELA L. HILPERT, ELLIS L. GESTEN, EMORY L. COWEN and WENDY E. SCHUBIN A B S T R A C T : T h e present study evaluated the effectiveness of a Social Problem Solving (SPS) competence training program for kindergartners, and examined relationships between SPS skill and adjustment gains. Subjects included 63 suburban middle-class Ss from three classes, who participated in the 42 lesson program, and 46 comparison Ss from two classes, who did not. Subjects were evaluated on problem solving, peer sociometric and teacher adjustment ratings. Program children gave significantly more, and better, solutions, and fewer irrelevant responses to interpersonal problems. They also improved more than comparison Ss on several teacher-rated dimensions of adjustment. Direct linkages between skill and adjustment gains, however, were not found.
In the past, mental health has emphasized a "casualty-repair" orientation; that is, its major efforts have been directed to undoing rooted maladjustment. Within that framework, resources have been insufficient to meet the need for psychological help (Albee, 1959, 1969}. That, along with other failings of the mental health delivery system {Cowen, 1980}, has directed attention to primary prevention alternatives such as promoting well-being and health, and competence training. In contrast to mental health's past restorative (therapeutic} emphasis, competence training is educational, mass-oriented and before-the-fact. Young children, because of their flexibility and modifiability, are seen as ideal targets for such programming (Cowen, 1977}. Social Problem Solving (SPS) Skills and Adjustment Relationships between interpersonal skills and adjustment have long been recognized. Early on, Jahoda (1953} defined interpersonal This research was supported by a grant from the New York State Department of Education, for which the authors express their appreciation. Janice I. Winer, Emory L. Cowen and Wendy E. Schubin are affiliated with the University of Rochester. Pamela Hilpert is affiliated with Albert Einstein Medical College, and Ellis Gesten with the University of South Florida. Requests for reprints should be mailed to the first author, Department of Psychology, the University of Rochester, River Campus Station, Rochester, NY 14627. 0278-095X(82)1400-0205502.75
205
© 1982 Human Sciences Press
206
Journal of Primary Prevention
problem solving in terms of three sequential strategies: admitting a problem, considering it, and taking effective action to resolve it. Mastery of those skills was, in her view, important to adjustment. A long line of subsequent research has shown that maladjusted or clinical samples at diverse ages and socioeconomic levels are deficient in a variety of interpersonal problem solving skills (Spivack & Levine, Note 6; Shure & Spivack, Note 2; Spivack, Platt, & Shure, 1976}. Such theoretical and empirical linkages have stimulated the development of competence training programs aimed at strengthening children's behavioral adjustment. One early prototype, Ojemann's (1961, 1967} causal thinking curriculum, was developed on the assumption that acquiring causal thinking skills would have adaptive as well as cognitive value, a view supported by the findings of several studies (Muuss, 1960; Griggs & Bonney, 1970}. A more recent conceptually related approach trains young children in a network of related interpersonal cognitive (or social} problemsolving skills, including problem identification, alternative sblution thinking, means-end thinking and consequential thinking (Spivack & Shure, 1974; Gesten, Flores de Apodaca, Rains, Weissberg & Cowen, 1979}. Alternative solution thinking, in particular, has been found to be a sensitive differentiator of young normal and maladjusted children IShure & Spivack, Note 3; Spivack et al., 1976}. Shure and Spivack's {Note 4} 46-lesson training program, for 4- and 5-year old disadvantaged inner-city children, was designed to impart basic ICPS skills and to improve behavioral adjustment. Based on a 2-year research evaluation of the program, they found that: (a} trained children gained significantly more than controls in ICPS skills, independent of IQ; (b) initially maladjusted Ss profited most from the program; {c} skill acquisition and adjustment gains were related; and (d) both acquisition and adjustment gains were maintained at a 1-year follow-up point (Shure & Spivack, 1978}. Similar findings were reported based on a program that trained inner-city mothers to teach problem-solving skills to their young children (Shure & Spivack, 1978}. However, a recent study (Krasnor & Rubin, Note 1} evaluating a modified version of Shure and Spivack's curriculum taught to middleclass preschoolers, failed to replicate the adjustment gains reported by those investigators. Other groups have developed comparable SPS training programs for older primary grade children (Elardo & Caldwell, 1979; Allen, Chinsky, Larcen, Lochman, & Selinger, 1976; Gesten et al., 1979}. Although each of those programs was able to show that trained children acquired the component SPS skills, none produced convincing
Winer, Hilpert, Gester~ Cowen, and Schubin
207
evidence of improved behavioral adjustment in target children. Two recent SPS program evaluation studies with inner-city and suburban 2nd-4th graders (Weissberg, Gesten, Rapkin, Cowen, Davidson, Flores de Apodaca, & McKim, in press; Weissberg, Gesten, Carnrike, Toro, Rapkin, Davidson, & Cowen, 1981} both demonstrated skill acquisition, including behavioral generalization of such acquisitions to a new and different situation, as well as improvements in several areas of teacher-rated adjustment. Neither study, however, could establish that acquisition actually led to improved behavioral adjustment. Thus, findings from a number of problem-solving interventions to date are consistent with respect to skill-acquisition, but not to adjustment. Children ranging from the preschool to the middlechildhood periods, with diverse sociodemographic backgrounds do indeed acquire such skills through training based on a variety of different curricula. Moreover, these "cognitive" gains generalize effectively to behavioral situations removed in time, place and person from the training setting (Weissberg, Gesten, Carnrike, et al., in press). The strongest evidence for post-intervention adjustive-gain (and its mediation by cognitive-gain) is with preschool, initially maladjusted, inner-city disadvantaged children (Spivack & Shure, 1974). Initially well-adjusted children, who have little "room" to improve, and older, suburban youngsters, show much weaker, if any, post-program adjustive gains. The extent to which cross-program differences in adjustment outcomes are attributable to sociodemographic factors, initial adjustment status variables, and/or to differences among SPS training programs remains unclear. Such information is needed to maximize the impact of future SPS training programs, and, thus, the long-term primary prevention contribution of this intriguing approach. The present study addresses that issue by developing, implementing, and evaluating an SPS program specifically designed for middle-income, suburban kindergartners. The study's two main questions are: (1) does the program lead to skill and adjustment gains? (2) are there relationships between those two sets of gains?
Method Subjects Subjects were kindergartners from five classes in three middle-income, predominantly white suburban schools. Experimental {program) Ss came from
208
Journal of Primary Prevention
the classes of three volunteer teachers in two schools that had previously had SPS training programs at the upper grade levels. Comparison Ss came from the classes of two kindergarten teachers in a demographically comparable school that had never had an SPS training program. Since classes could not be assigned randomly to conditions, a non-equivalent control design was used. Based on the complete preprogram problem solving and adjustment data, the total N for the study was 109--63 experimental Ss (33 boys and 30 girls) and 46 comparison Ss (24 boys and 22 girls). Since several children either moved or were absent at post testing, the final N for program evaluation purposes was 102--57 experimental Ss and 45 comparison Ss.
Procedure Pretesting was done in January, 1979, and posttesting in May, 1979, several weeks after the program ended. Trained undergraduate evaluators administered all tests given to children. Adjustment ratings were submitted by teachers. Verbal IQ scores were obtained from school records.
S t u d e n t aide/evaluators Six undergraduate aides were selected and trained as program aides and evaluators. They were chosen on the basis of interest and prior experience from 25 respondents to an advertisement in the campus paper. Evaluators were taught to administer the criterion tests in two 4-hour training sessions which preceded their training in program content. Four new undergraduate evaluators were recruited and trained for posttesting. Two of the original evaluators, who also did posttesting, were assigned to classes other than those in which they had served as aides.
Curriculum The program curriculum, adapted from earlier ones {Gesten et al., 1979; Shure & Spivack, Note 4}, differed from Shure and Spivack's {Note 4) in several key respects. For one thing, because it was targeted to suburban children with relatively well-developed language skills, it deemphasized {i.e., from 23 to 7 lessons} training in prerequisite language skills, thus allowing for more time for training substantive SPS skills. Second, the training structure was streamlined and built around a simplified three-step strategy for approaching interpersonal problems: (1)"Say the problem"; (2}"Think of lots of things to do"; and (3)"Think of what might happen next". Role-playing was used extensively to concretize component skill acquisition. A 130-page training curriculum developed specifically for the program, included 42 "lessons" covering five main units: (a) prerequisite language skills--7, (b) feeling identification--10, (c) problem identification--7, (d) generation of alternative solutions--8, and (e} consequential thinking and
Winer, Hilpert, Gesten, Cowen, and Schubin
209
integration-10. The curriculum was highly structured with detailed scripts for teaching each program skill. Formal lessons were enriched by participative games and activities involving visual, auditory and motor modalities. Examples of such activities include group discussions, puppet shows, posterpictorials of problems and feelings, and cartoon workbook exercises. Lessons were taught four times a week for 10 weeks, starting in January, 1979, and ending in April, 1979. Teachers and aides were given curriculum units about 10 days before each unit started. Both groups attended weekly, 1-1/2 hour training sessions to discuss and role-play the week's lessons. There were also biweekly individual consulting sessions with teachers to review program developments and/or concerns. Aides attended a separate weekly 1-1/2 hour class to role-play lessons, discuss class management problems, and to prepare curriculum materials.
Measures SPS Skills: Kindergarten Alternative Solution Measure (KASM). The KASM, a modification of the Preschool Interpersonal Problem Solving (PIPS) Test (Shure & Spivack, Note 5), was used to assess children's problem-solving skills. This individually administered measure consists of two illustrated story pairs: peer and adult. The peer stories depict a child who wants a toy that another child is using. The adult stories show a child damaging something that belongs to his/her mother.The second story of both sets depicts the same problem as the first, only the concrete objects and characters change. The task in each case is to say all the different things that the protagonist might do to solve his/her problem. Responses to the four stories were scored in four content categories and for effectiveness. The content categories included: (a) alternative solutions--novel actions by the protagonist in response to the problem, and directed toward the specified goal; (b) solution variants--variations of a previous alternative solution; (c) solution repetitions--verbatim repetitions or rephrasings of prior responses; and {d) irrelevant responses--responses in which the protagonist takes no meaningful action to solve the problem. Alternative solutions were also evaluated on a 5-point effectiveness scale Imaximal=5, minimal----l) based on pooled a priori judgments by five independent raters about the extent to which a solution maximized positive consequences, considered long-term effects, and was feasible for a 5-year old. KASM data were analyzed for the above prime variables and two derivative ones: (1) total solutions--the sum of alternative solutions and solution variants, that is, "goal-directed" solutions, and (2} the number of alternative solutions with effectiveness ratings>3. Two undergraduate raters and one criterion judge scored the KASM. Separate training rounds were run until the raters reached an average agreement of 85% with the criterion decision. The 204 KASM protocols (pre and post) were scored in four rounds; on each round 20% were used as reliability protocols. Category percent agreements between the two raters and
210
Journal of Primary Prevention
the criterion decision ranged from .82-.96. Overall percent agreements Ibased on alternative solutions, solution variants, repetitions and irrelevant responses} ranged from .88-.94. Effectiveness ratings, using the same procedures, raters and criterion judge, yielded reliability coefficients of from .90-.99.
Adjustment Teacher ratings. Teachers rated Ss' classroom adjustment on the Rochester Kindergarten Behavior Rating Scale (RKBRS), derived from two prior measures: (1} the Classroom Adjustment Ratings Scale--CARS (Lorion, Cowen, & Caldwell, 1975}, and (2} the Health Resources I n v e n t o r y - - H R I (Gesten, 1976). Both measures have been shown to have satisfactory {mid .80s} reliability. The two subsections of the behavioral ratings scale included actual CARS and H R I items plus several new items {e.g., persists in the face of obstacles; accepts responsibility for his/her own actions} reflecting program-relevant behaviors. Section 1 {12 items} assessed nonacademic problem behaviors {e.g., impulsive, often unable to wait; easily upset} whereas Section 2 (16 items} assessed competencies (e.g., feels good about self; expresses ideas willingly}. Teachers rated all items on 5-point scales with higher ratings reflecting more serious school problems or greater strengths. Based on pretest data, 1 separate principal components analyses with varimax rotations yielded two factors for Section 1 and three for Section 2, accounting respectively for 64% and 59% of the total variance. The two problem dimensions were: {1) acting-out--aggressive impulsive behaviors; (2} shy-anxious--withdrawn, uncommunicative, tense behaviors. Also derived was a problem total, a sum of the two factors. Competence factors included: {1) general social maturity--responsibility and thoughtfulness; (2} frustrationtolerance--ability to cope with failure; {3} gutsy-assertiveness--ability to express needs and feelings appropriately. A competence total, the sum of those three competence factors, was also used. Sociometric Ratings. This measure, given in small IN = 5-7) groups, modified one used by Gesten et al. 11979}. Children rated all classmates on 3-point scales by placing an " X " on faces drawn to represent the following statements: {1) I t ' s OK/{2) I ' m happy that/{3) I ' m very, very happy t h a t / " X " is in my class. Two scores were derived: {1} the mean ratings that each S assigned to his/her classmates Isent mean}, and {2} the mean ratings assigned to each S by classmates {received mean}. Pilot testing showed that kindergarten children both understood the task and gave adequately distributed ratings. 1Data for eight other children, who did not take the problem-solving tests, were included in these analyses, bringing the total N to 117.
Winer, Hilper~ Gesten, Cowen, and Schubin
211
IQ Data Peabody Picture Vocabulary Test scores (Dunn, 1965) were obtained from school records.
Results The results are presented in two main sections: (1} the effects of SPS training on problem solving and adjustment; and (2) relationships between problem solving and adjustment change-scores for program children.
Effects of SPS Training on Problem Solving and Adjustment Group differences in pre-post change scores for all acquisition and adjustment measures were tested separately using 2(sex) X 2(condition) X 5(teacher) nested ANOVAs for all 15 (6 KASM, 9 adjustment) dependent measures. Condition and sex were between Ss' effects with teacher nested within condition. Descriptive data for pre- and change-scores, by group, and Fs for the change (A) scores' analyses are presented in Table 1. Unless otherwise indicated, high scores indicate better performance, and positive change-scores reflect improvement. T-tests indicated that the two groups were comparable on 13 of the 15 criteria prescores, and on IQ. The exceptions were that experimentals gave somewhat more irrelevant responses than comparison Ss (t = 1.95, p < .06) and were rated as less well adjusted on the shy-anxious factor (t = 2.34, p 3. They also improved more on two adjustment measures: problem total and frustration-tolerance, with directionally similar trends (ps -- .08-.13) on three others: acting-out, shy-anxious and competence total. On shyanxious and problem total, however, the difference between groups may have been due, in part, to significant nested teacher-effects. There were no differences between groups in changes on the two sociometric measures. Correlations were computed between the six SPS and the nine adjustment change scores, for 57 program Ss. None of the 54 rs was significant. Since such rs, in principle, might have been restricted by
Journal of Primary Prevention
212
.o
~u
o
,~
~ , ~
U
~
~"
~o
®
b
L, ~, b ~ ~
u,
~
o C
. . . . . . . . . .
•
~,~
~
~
~
~., ~ ,
~., . ~ ~
.-
~,~
g
~o
oo ~ . . . . . . . . . . . . . . . . . . . . . . k, b ~, ;~ k, ~,
. . . . . . . . .
L~
o"
~ b ~ 0
g
•
.
.
. ~ ? o . o
.
.
.
.
.
. C'~
0
0 w
.
.
.
.
.
o0
.
.
.
.
.
.
.
•
n n
~ o ~ a
i, ~
i.-~ i . - i ~-~
Winer, Hilpert, Geste~ Cowen, and Schubin
213
the inclusion of initially well adjusted Ss, who had "no room to improve" on the adjustment measures, they were recomputed. The new computations used an increase in alternate solutions as the criterion, both for change in acting-out {excluding 22 Ss with no or very few initial acting-out problems}, and for change in shy-anxious behavior {excluding the 27 Ss with no or few initial shy-anxious problems}. In neither case did the new. r based on the curtailed group of Ss approach significance. Discussion
The study's most basic findings were that experimental Ss improved significantly more than comparison Ss in all aspects of social problem solving, the program's most direct focus, and in several areas of adjustment. Those findings are reasonably consistent with Spivack and Shure's data {1974} based on young inner-city children, but they contrast, for adjustment, with those reported by Krasnor and Rubin {Note 1) for middle-class preschoolers. Post-training skill-gains evidenced by program children were substantial absolutely. Their average gain of nearly three alternative solutions increased preprogram response levels by 75%, and their average gain of 10 solution variants increased preprogram response levels for that category by more than 200%. Moreover, their post-solutions were qualitatively superior {i.e., rated by judges as more likely to lead to effective resolutions of interpersonal problems} both to comparison Ss and to their own preprogram responses. In several respects, the present acquisition findings are stronger than those reported by Krasnor and Rubin {Note 1) based on a similar program and a similar suburban sample. Outcome differences between the two studies may be curriculum-related. Whereas Krasnor and Rubin used an abridged version of Shure and Spivack's {Note 4) curriculum developed for inner-city preschoolers, the present program modified that curriculum substantially by: a) spending less time on prerequisite language skills and more on substantive program skills, b) highlighting a streamlined functional set of problem-solving steps attuned to a kindergartner's developmental level, and c) making it more interesting and ego-involving through extensive use of roleplaying and a rich array of age-relevant games and activities. The present study's strong skill-acquisition findings suggest that these curricular changes were appropriate to the needs, styles and prior learning patterns of middle-income suburban children.
214
Journal of Primary Prevention
One shortcoming of this study is that it did not include an outside behavioral test of the generalization of skill acquisition, primarily because of the complexities involved in developing such a paradigm for very young children. Behavioral generalization of acquired SPS cognitive skills has, however, been reported for older suburban samples {Allen et al., 1976, Weissberg, Gesten, Carnrike, et al., 1981}. Both of those samples offered less compelling evidence of skill acquisition than did the present kindergarten group. Improvements by program Ss in teacher-rated adjustment were reasonably consistent with prior findings by Spivack and Shure {1974L After training, teachers judged program children to have fewer adjustment problems than comparison Ss, and to have gained more in competence, especially in frustration-tolerance, a competence that relates directly to the intervention's main goals. Generalization on the basis of those findings, however, is limited by several of the study's realities. Thus, the study lacks a true control group, in that the program teachers were volunteers who were aware of their classes' program/nonprogram status. Coupled with that is the fact that the adjustment measures that differentiated program and comparison Ss were teacher-ratings, not peer-ratings. It could be argued that teachers' involvement in the program may have biased their adjustment ratings in favor of program children. For that reason, we interpret the adjustment findings as, at best, suggestive. Additional studies of this type, better controlled in terms of teacher-selection and with a broader range of adjustment criteria, are needed to clarify the present findings. The absence of direct links between improvements in SPS skills and adjustment, contrasts with Spivack and Shure's {1974} consistent demonstration of such relationships. Their work, however, was done with disadvantaged preschoolers, many of whom were identified as maladjusted before training. Among middle-class kindergartners, relatively few of whom were identified initially as maladjusted, gains in alternative solution thinking did not mediate improved adjustment. In that sense, current findings bearing on relationships between skillgain and adjustment-gain more closely approximate those reported by programs with older middle-SES children (e.g., Gesten et al., 1979; Elardo & Caldwell, 1979; Weissberg, Gesten, Rapkin, et al., 1981}. Although the present Es improved both in problem-solving skills and on teacher ratings of adjustment, apparently different children gained in those two sectors, and skill-gain did not specifically mediate adjustment gain. That may be due either to intrinsic limits on the gain potential of many Ss, because they were relatively well-adjusted initially, or to the fact that problem-solving skills, other than those
Winer, Hilpert, Gester~ Cowen, and Schubin
215
tapped by this study, will ultimately prove to be key adjustment mediators for this age/SES group. In summary, although the present SPS training program strengthened children's problem-solving skills considerably, and to a lesser extent, aspects of their adjustment, it failed to show direct effects of SPS skill acquisition on adjustive gain for middle-SES kindergartners. That failure is vexing, because such a link undergirds primary prevention's rationale for training competencies in young children. The need to understand better the active agents in the mediation of adjustment-gain remains an important prerequisite for the robust development of future primary prevention work with young children.
Reference Notes 1. Krasnor, R.L., & Rubin, K.H. Preschoolers' verbal and behavioral solutions to social problems. Paper presented at Canadian Psychological Association, Ottawa, Ontario, June, 1978. 2. Shure, M.B., & Spivack, G. Cognitiveproblem-solving skills, adjustment and social class. Research and Evaluation Report. Philadelphia: Department of Mental Health Sciences, Hahnemann Medical College and Hospital, 1970. 3. Shure, M.B., & Spivack, G. Problem-solving capacity, social class and adjustment among nursery school children. Paper p r e s e n t e d at E a s t e r n Psychological Association, Atlantic City, New Jersey, 1970. 4. Shure, M.B., & Spivack, G. A mental health program for kindergarten children: Training script. Philadelphia: Department of Mental Health Sciences, Hahnemann Community Mental Health/Mental Retardation Center, 1974. 5. Shure, M.B., & Spivack, G. Preschool Interpersonal Problem Solving (PIPS] Test: Manual. Philadelphia: Department of Mental Health Sciences, Hahnemann Community Mental Health/Mental Retardation Center, 1974. 6. Spivack, G., & Levine, M. Self-regulation in acting-out and normal adolescents. Report M-4531, National Institute of Health, Washington, D.C., 1963.
References Albee, G.W. Mental health manpower trends. New York: Basic Books, 1959. Albee, G.W. The relation of conceptual models of disturbed behavior to institutional and manpower requirements. In F.N. Arhnoff, E.A. Rubenstein, & J.S. Speisman (Eds.), Manpower for mental health. Chicago: Aldine, 1969. Allen, G.J., Chinsky, J.M., Larcen, S.W., Lochman, J.E., & Selinger, H.V. Community psychology and the schools: A behaviorally oriented multilevel preventive approach. Hillsdale, N.J.: Lawrence Erlbaum Associates, 1976. Cowen, E.L. Baby-steps toward primary prevention. American Journal of Community Psychology, 1977, 5, 1-22. Cowen, E.L. The wooing of primary prevention. American Journal of Community Psychology, 1980, 8, 258-284. Dunn, L.M. Peabody Picture Vocabulary Test: Manual. Minnesota: American Guidance Service, Inc., 1965.
216
Journal of Primary Prevention
Elardo, P.T., & Caldwell, B.M. The effects of an experimental social development program on children in the middle childhood period. Psychology in the Schools, 1979, 16, 93-100. Gesten, E.L. A Health Resources Inventory: The development of a measure of the personal and social competence of primary-grade children. Journal of Consulting and Clinical Psychology, 1976, 44, 775-786. Gesten, E.L., Flores de Apodaca, R., Rains, M., Weissberg, R.P., & Cowen, E.L. Promoting peer related social competence in school. In M.W. Kent & J.E. Rolf (Eds.), The primary prevention of psychopathology. Volume 3: Promoting social competence and coping in children. Hanover, N.H.: University Press of New England, 1979. Griggs, J.W., & Bonney, M.E. Relationship between "causal" orientation and acceptance of others, "self-ideal self-congruence," and mental health changes for fourth and fifth grade children. Journal of Educational Research, 1970, 63, 471-477. Jahoda, M. The meaning of psychological health. Social Casework, 1953, 34, 349-354. Lorion, R.P., Cowen, E.L., & Caldwell, R.A. Normative data and parametric analyses of school maladjustment. American Journal of Community Psychology, 1975, 3, 291-301. Muuss, R.E. The effects of a one and two year causal learning program. Journal of Personality, 1960, 28, 479-491. Ojemann, R.H. Investigations on the effects of teacher understanding and appreciation of behavior dynamics. In G. Caplan (Ed.), Prevention of mental disorders in children. New York: Basic Books, 1961. Ojemann, R.H. Incorporating psychological concepts in the school curriculum. Journal of School Psychology, 1967, 5, 195-204. Shure, M.B., & Spivack, G. Problem solving techniques in childrearing. San Francisco: Jossey-Bass, 1978. Spivack, G., Platt, J.J., & Shure, M.B. The problem-solving approach to adjustment. San Francisco: Jossey-Bass, 1976. Spivack, G., & Shure, M.B. Social adjustment of young children: A cognitive approach to solving real life problems. San Francisco: Jossey-Bass, 1974. Weissberg, R.P., Gesten, E.L., Carnrike, C.L., Toro, P.A., Rapkin, B.D., Davidson, E., & Cowen, E.L. Social problem solving skills training: A competence-building intervention with 2nd-4th grade children, American Journal of Community Psychology, 1981, 9, 411-424. Weissberg, R.P., Gesten, E.L., Rapkin, B.D., Cowen, E.L., Davidson, E., Flores de Apodaca, R., & McKim, B. Evaluation of social-problem-solving training for suburban and inner-city third-grade children. Journal of Consulting and Clinical Psychology, 1981, 49, 251-261.
The Evaluation of a Kindergarten Social Problem Solving Program JANICE I. WINER, PAMELA L. HILPERT, ELLIS L. GESTEN, EMORY L. COWEN and WENDY E. SCHUBIN A B S T R A C T : T h e present study evaluated the effectiveness of a Social Problem Solving (SPS) competence training program for kindergartners, and examined relationships between SPS skill and adjustment gains. Subjects included 63 suburban middle-class Ss from three classes, who participated in the 42 lesson program, and 46 comparison Ss from two classes, who did not. Subjects were evaluated on problem solving, peer sociometric and teacher adjustment ratings. Program children gave significantly more, and better, solutions, and fewer irrelevant responses to interpersonal problems. They also improved more than comparison Ss on several teacher-rated dimensions of adjustment. Direct linkages between skill and adjustment gains, however, were not found.
In the past, mental health has emphasized a "casualty-repair" orientation; that is, its major efforts have been directed to undoing rooted maladjustment. Within that framework, resources have been insufficient to meet the need for psychological help (Albee, 1959, 1969}. That, along with other failings of the mental health delivery system {Cowen, 1980}, has directed attention to primary prevention alternatives such as promoting well-being and health, and competence training. In contrast to mental health's past restorative (therapeutic} emphasis, competence training is educational, mass-oriented and before-the-fact. Young children, because of their flexibility and modifiability, are seen as ideal targets for such programming (Cowen, 1977}. Social Problem Solving (SPS) Skills and Adjustment Relationships between interpersonal skills and adjustment have long been recognized. Early on, Jahoda (1953} defined interpersonal This research was supported by a grant from the New York State Department of Education, for which the authors express their appreciation. Janice I. Winer, Emory L. Cowen and Wendy E. Schubin are affiliated with the University of Rochester. Pamela Hilpert is affiliated with Albert Einstein Medical College, and Ellis Gesten with the University of South Florida. Requests for reprints should be mailed to the first author, Department of Psychology, the University of Rochester, River Campus Station, Rochester, NY 14627. 0278-095X(82)1400-0205502.75
205
© 1982 Human Sciences Press
206
Journal of Primary Prevention
problem solving in terms of three sequential strategies: admitting a problem, considering it, and taking effective action to resolve it. Mastery of those skills was, in her view, important to adjustment. A long line of subsequent research has shown that maladjusted or clinical samples at diverse ages and socioeconomic levels are deficient in a variety of interpersonal problem solving skills (Spivack & Levine, Note 6; Shure & Spivack, Note 2; Spivack, Platt, & Shure, 1976}. Such theoretical and empirical linkages have stimulated the development of competence training programs aimed at strengthening children's behavioral adjustment. One early prototype, Ojemann's (1961, 1967} causal thinking curriculum, was developed on the assumption that acquiring causal thinking skills would have adaptive as well as cognitive value, a view supported by the findings of several studies (Muuss, 1960; Griggs & Bonney, 1970}. A more recent conceptually related approach trains young children in a network of related interpersonal cognitive (or social} problemsolving skills, including problem identification, alternative sblution thinking, means-end thinking and consequential thinking (Spivack & Shure, 1974; Gesten, Flores de Apodaca, Rains, Weissberg & Cowen, 1979}. Alternative solution thinking, in particular, has been found to be a sensitive differentiator of young normal and maladjusted children IShure & Spivack, Note 3; Spivack et al., 1976}. Shure and Spivack's {Note 4} 46-lesson training program, for 4- and 5-year old disadvantaged inner-city children, was designed to impart basic ICPS skills and to improve behavioral adjustment. Based on a 2-year research evaluation of the program, they found that: (a} trained children gained significantly more than controls in ICPS skills, independent of IQ; (b) initially maladjusted Ss profited most from the program; {c} skill acquisition and adjustment gains were related; and (d) both acquisition and adjustment gains were maintained at a 1-year follow-up point (Shure & Spivack, 1978}. Similar findings were reported based on a program that trained inner-city mothers to teach problem-solving skills to their young children (Shure & Spivack, 1978}. However, a recent study (Krasnor & Rubin, Note 1} evaluating a modified version of Shure and Spivack's curriculum taught to middleclass preschoolers, failed to replicate the adjustment gains reported by those investigators. Other groups have developed comparable SPS training programs for older primary grade children (Elardo & Caldwell, 1979; Allen, Chinsky, Larcen, Lochman, & Selinger, 1976; Gesten et al., 1979}. Although each of those programs was able to show that trained children acquired the component SPS skills, none produced convincing
Winer, Hilpert, Gester~ Cowen, and Schubin
207
evidence of improved behavioral adjustment in target children. Two recent SPS program evaluation studies with inner-city and suburban 2nd-4th graders (Weissberg, Gesten, Rapkin, Cowen, Davidson, Flores de Apodaca, & McKim, in press; Weissberg, Gesten, Carnrike, Toro, Rapkin, Davidson, & Cowen, 1981} both demonstrated skill acquisition, including behavioral generalization of such acquisitions to a new and different situation, as well as improvements in several areas of teacher-rated adjustment. Neither study, however, could establish that acquisition actually led to improved behavioral adjustment. Thus, findings from a number of problem-solving interventions to date are consistent with respect to skill-acquisition, but not to adjustment. Children ranging from the preschool to the middlechildhood periods, with diverse sociodemographic backgrounds do indeed acquire such skills through training based on a variety of different curricula. Moreover, these "cognitive" gains generalize effectively to behavioral situations removed in time, place and person from the training setting (Weissberg, Gesten, Carnrike, et al., in press). The strongest evidence for post-intervention adjustive-gain (and its mediation by cognitive-gain) is with preschool, initially maladjusted, inner-city disadvantaged children (Spivack & Shure, 1974). Initially well-adjusted children, who have little "room" to improve, and older, suburban youngsters, show much weaker, if any, post-program adjustive gains. The extent to which cross-program differences in adjustment outcomes are attributable to sociodemographic factors, initial adjustment status variables, and/or to differences among SPS training programs remains unclear. Such information is needed to maximize the impact of future SPS training programs, and, thus, the long-term primary prevention contribution of this intriguing approach. The present study addresses that issue by developing, implementing, and evaluating an SPS program specifically designed for middle-income, suburban kindergartners. The study's two main questions are: (1) does the program lead to skill and adjustment gains? (2) are there relationships between those two sets of gains?
Method Subjects Subjects were kindergartners from five classes in three middle-income, predominantly white suburban schools. Experimental {program) Ss came from
208
Journal of Primary Prevention
the classes of three volunteer teachers in two schools that had previously had SPS training programs at the upper grade levels. Comparison Ss came from the classes of two kindergarten teachers in a demographically comparable school that had never had an SPS training program. Since classes could not be assigned randomly to conditions, a non-equivalent control design was used. Based on the complete preprogram problem solving and adjustment data, the total N for the study was 109--63 experimental Ss (33 boys and 30 girls) and 46 comparison Ss (24 boys and 22 girls). Since several children either moved or were absent at post testing, the final N for program evaluation purposes was 102--57 experimental Ss and 45 comparison Ss.
Procedure Pretesting was done in January, 1979, and posttesting in May, 1979, several weeks after the program ended. Trained undergraduate evaluators administered all tests given to children. Adjustment ratings were submitted by teachers. Verbal IQ scores were obtained from school records.
S t u d e n t aide/evaluators Six undergraduate aides were selected and trained as program aides and evaluators. They were chosen on the basis of interest and prior experience from 25 respondents to an advertisement in the campus paper. Evaluators were taught to administer the criterion tests in two 4-hour training sessions which preceded their training in program content. Four new undergraduate evaluators were recruited and trained for posttesting. Two of the original evaluators, who also did posttesting, were assigned to classes other than those in which they had served as aides.
Curriculum The program curriculum, adapted from earlier ones {Gesten et al., 1979; Shure & Spivack, Note 4}, differed from Shure and Spivack's {Note 4) in several key respects. For one thing, because it was targeted to suburban children with relatively well-developed language skills, it deemphasized {i.e., from 23 to 7 lessons} training in prerequisite language skills, thus allowing for more time for training substantive SPS skills. Second, the training structure was streamlined and built around a simplified three-step strategy for approaching interpersonal problems: (1)"Say the problem"; (2}"Think of lots of things to do"; and (3)"Think of what might happen next". Role-playing was used extensively to concretize component skill acquisition. A 130-page training curriculum developed specifically for the program, included 42 "lessons" covering five main units: (a) prerequisite language skills--7, (b) feeling identification--10, (c) problem identification--7, (d) generation of alternative solutions--8, and (e} consequential thinking and
Winer, Hilpert, Gesten, Cowen, and Schubin
209
integration-10. The curriculum was highly structured with detailed scripts for teaching each program skill. Formal lessons were enriched by participative games and activities involving visual, auditory and motor modalities. Examples of such activities include group discussions, puppet shows, posterpictorials of problems and feelings, and cartoon workbook exercises. Lessons were taught four times a week for 10 weeks, starting in January, 1979, and ending in April, 1979. Teachers and aides were given curriculum units about 10 days before each unit started. Both groups attended weekly, 1-1/2 hour training sessions to discuss and role-play the week's lessons. There were also biweekly individual consulting sessions with teachers to review program developments and/or concerns. Aides attended a separate weekly 1-1/2 hour class to role-play lessons, discuss class management problems, and to prepare curriculum materials.
Measures SPS Skills: Kindergarten Alternative Solution Measure (KASM). The KASM, a modification of the Preschool Interpersonal Problem Solving (PIPS) Test (Shure & Spivack, Note 5), was used to assess children's problem-solving skills. This individually administered measure consists of two illustrated story pairs: peer and adult. The peer stories depict a child who wants a toy that another child is using. The adult stories show a child damaging something that belongs to his/her mother.The second story of both sets depicts the same problem as the first, only the concrete objects and characters change. The task in each case is to say all the different things that the protagonist might do to solve his/her problem. Responses to the four stories were scored in four content categories and for effectiveness. The content categories included: (a) alternative solutions--novel actions by the protagonist in response to the problem, and directed toward the specified goal; (b) solution variants--variations of a previous alternative solution; (c) solution repetitions--verbatim repetitions or rephrasings of prior responses; and {d) irrelevant responses--responses in which the protagonist takes no meaningful action to solve the problem. Alternative solutions were also evaluated on a 5-point effectiveness scale Imaximal=5, minimal----l) based on pooled a priori judgments by five independent raters about the extent to which a solution maximized positive consequences, considered long-term effects, and was feasible for a 5-year old. KASM data were analyzed for the above prime variables and two derivative ones: (1) total solutions--the sum of alternative solutions and solution variants, that is, "goal-directed" solutions, and (2} the number of alternative solutions with effectiveness ratings>3. Two undergraduate raters and one criterion judge scored the KASM. Separate training rounds were run until the raters reached an average agreement of 85% with the criterion decision. The 204 KASM protocols (pre and post) were scored in four rounds; on each round 20% were used as reliability protocols. Category percent agreements between the two raters and
210
Journal of Primary Prevention
the criterion decision ranged from .82-.96. Overall percent agreements Ibased on alternative solutions, solution variants, repetitions and irrelevant responses} ranged from .88-.94. Effectiveness ratings, using the same procedures, raters and criterion judge, yielded reliability coefficients of from .90-.99.
Adjustment Teacher ratings. Teachers rated Ss' classroom adjustment on the Rochester Kindergarten Behavior Rating Scale (RKBRS), derived from two prior measures: (1} the Classroom Adjustment Ratings Scale--CARS (Lorion, Cowen, & Caldwell, 1975}, and (2} the Health Resources I n v e n t o r y - - H R I (Gesten, 1976). Both measures have been shown to have satisfactory {mid .80s} reliability. The two subsections of the behavioral ratings scale included actual CARS and H R I items plus several new items {e.g., persists in the face of obstacles; accepts responsibility for his/her own actions} reflecting program-relevant behaviors. Section 1 {12 items} assessed nonacademic problem behaviors {e.g., impulsive, often unable to wait; easily upset} whereas Section 2 (16 items} assessed competencies (e.g., feels good about self; expresses ideas willingly}. Teachers rated all items on 5-point scales with higher ratings reflecting more serious school problems or greater strengths. Based on pretest data, 1 separate principal components analyses with varimax rotations yielded two factors for Section 1 and three for Section 2, accounting respectively for 64% and 59% of the total variance. The two problem dimensions were: {1) acting-out--aggressive impulsive behaviors; (2} shy-anxious--withdrawn, uncommunicative, tense behaviors. Also derived was a problem total, a sum of the two factors. Competence factors included: {1) general social maturity--responsibility and thoughtfulness; (2} frustrationtolerance--ability to cope with failure; {3} gutsy-assertiveness--ability to express needs and feelings appropriately. A competence total, the sum of those three competence factors, was also used. Sociometric Ratings. This measure, given in small IN = 5-7) groups, modified one used by Gesten et al. 11979}. Children rated all classmates on 3-point scales by placing an " X " on faces drawn to represent the following statements: {1) I t ' s OK/{2) I ' m happy that/{3) I ' m very, very happy t h a t / " X " is in my class. Two scores were derived: {1} the mean ratings that each S assigned to his/her classmates Isent mean}, and {2} the mean ratings assigned to each S by classmates {received mean}. Pilot testing showed that kindergarten children both understood the task and gave adequately distributed ratings. 1Data for eight other children, who did not take the problem-solving tests, were included in these analyses, bringing the total N to 117.
Winer, Hilper~ Gesten, Cowen, and Schubin
211
IQ Data Peabody Picture Vocabulary Test scores (Dunn, 1965) were obtained from school records.
Results The results are presented in two main sections: (1} the effects of SPS training on problem solving and adjustment; and (2) relationships between problem solving and adjustment change-scores for program children.
Effects of SPS Training on Problem Solving and Adjustment Group differences in pre-post change scores for all acquisition and adjustment measures were tested separately using 2(sex) X 2(condition) X 5(teacher) nested ANOVAs for all 15 (6 KASM, 9 adjustment) dependent measures. Condition and sex were between Ss' effects with teacher nested within condition. Descriptive data for pre- and change-scores, by group, and Fs for the change (A) scores' analyses are presented in Table 1. Unless otherwise indicated, high scores indicate better performance, and positive change-scores reflect improvement. T-tests indicated that the two groups were comparable on 13 of the 15 criteria prescores, and on IQ. The exceptions were that experimentals gave somewhat more irrelevant responses than comparison Ss (t = 1.95, p < .06) and were rated as less well adjusted on the shy-anxious factor (t = 2.34, p 3. They also improved more on two adjustment measures: problem total and frustration-tolerance, with directionally similar trends (ps -- .08-.13) on three others: acting-out, shy-anxious and competence total. On shyanxious and problem total, however, the difference between groups may have been due, in part, to significant nested teacher-effects. There were no differences between groups in changes on the two sociometric measures. Correlations were computed between the six SPS and the nine adjustment change scores, for 57 program Ss. None of the 54 rs was significant. Since such rs, in principle, might have been restricted by
Journal of Primary Prevention
212
.o
~u
o
,~
~ , ~
U
~
~"
~o
®
b
L, ~, b ~ ~
u,
~
o C
. . . . . . . . . .
•
~,~
~
~
~
~., ~ ,
~., . ~ ~
.-
~,~
g
~o
oo ~ . . . . . . . . . . . . . . . . . . . . . . k, b ~, ;~ k, ~,
. . . . . . . . .
L~
o"
~ b ~ 0
g
•
.
.
. ~ ? o . o
.
.
.
.
.
. C'~
0
0 w
.
.
.
.
.
o0
.
.
.
.
.
.
.
•
n n
~ o ~ a
i, ~
i.-~ i . - i ~-~
Winer, Hilpert, Geste~ Cowen, and Schubin
213
the inclusion of initially well adjusted Ss, who had "no room to improve" on the adjustment measures, they were recomputed. The new computations used an increase in alternate solutions as the criterion, both for change in acting-out {excluding 22 Ss with no or very few initial acting-out problems}, and for change in shy-anxious behavior {excluding the 27 Ss with no or few initial shy-anxious problems}. In neither case did the new. r based on the curtailed group of Ss approach significance. Discussion
The study's most basic findings were that experimental Ss improved significantly more than comparison Ss in all aspects of social problem solving, the program's most direct focus, and in several areas of adjustment. Those findings are reasonably consistent with Spivack and Shure's data {1974} based on young inner-city children, but they contrast, for adjustment, with those reported by Krasnor and Rubin {Note 1) for middle-class preschoolers. Post-training skill-gains evidenced by program children were substantial absolutely. Their average gain of nearly three alternative solutions increased preprogram response levels by 75%, and their average gain of 10 solution variants increased preprogram response levels for that category by more than 200%. Moreover, their post-solutions were qualitatively superior {i.e., rated by judges as more likely to lead to effective resolutions of interpersonal problems} both to comparison Ss and to their own preprogram responses. In several respects, the present acquisition findings are stronger than those reported by Krasnor and Rubin {Note 1) based on a similar program and a similar suburban sample. Outcome differences between the two studies may be curriculum-related. Whereas Krasnor and Rubin used an abridged version of Shure and Spivack's {Note 4) curriculum developed for inner-city preschoolers, the present program modified that curriculum substantially by: a) spending less time on prerequisite language skills and more on substantive program skills, b) highlighting a streamlined functional set of problem-solving steps attuned to a kindergartner's developmental level, and c) making it more interesting and ego-involving through extensive use of roleplaying and a rich array of age-relevant games and activities. The present study's strong skill-acquisition findings suggest that these curricular changes were appropriate to the needs, styles and prior learning patterns of middle-income suburban children.
214
Journal of Primary Prevention
One shortcoming of this study is that it did not include an outside behavioral test of the generalization of skill acquisition, primarily because of the complexities involved in developing such a paradigm for very young children. Behavioral generalization of acquired SPS cognitive skills has, however, been reported for older suburban samples {Allen et al., 1976, Weissberg, Gesten, Carnrike, et al., 1981}. Both of those samples offered less compelling evidence of skill acquisition than did the present kindergarten group. Improvements by program Ss in teacher-rated adjustment were reasonably consistent with prior findings by Spivack and Shure {1974L After training, teachers judged program children to have fewer adjustment problems than comparison Ss, and to have gained more in competence, especially in frustration-tolerance, a competence that relates directly to the intervention's main goals. Generalization on the basis of those findings, however, is limited by several of the study's realities. Thus, the study lacks a true control group, in that the program teachers were volunteers who were aware of their classes' program/nonprogram status. Coupled with that is the fact that the adjustment measures that differentiated program and comparison Ss were teacher-ratings, not peer-ratings. It could be argued that teachers' involvement in the program may have biased their adjustment ratings in favor of program children. For that reason, we interpret the adjustment findings as, at best, suggestive. Additional studies of this type, better controlled in terms of teacher-selection and with a broader range of adjustment criteria, are needed to clarify the present findings. The absence of direct links between improvements in SPS skills and adjustment, contrasts with Spivack and Shure's {1974} consistent demonstration of such relationships. Their work, however, was done with disadvantaged preschoolers, many of whom were identified as maladjusted before training. Among middle-class kindergartners, relatively few of whom were identified initially as maladjusted, gains in alternative solution thinking did not mediate improved adjustment. In that sense, current findings bearing on relationships between skillgain and adjustment-gain more closely approximate those reported by programs with older middle-SES children (e.g., Gesten et al., 1979; Elardo & Caldwell, 1979; Weissberg, Gesten, Rapkin, et al., 1981}. Although the present Es improved both in problem-solving skills and on teacher ratings of adjustment, apparently different children gained in those two sectors, and skill-gain did not specifically mediate adjustment gain. That may be due either to intrinsic limits on the gain potential of many Ss, because they were relatively well-adjusted initially, or to the fact that problem-solving skills, other than those
Winer, Hilpert, Gester~ Cowen, and Schubin
215
tapped by this study, will ultimately prove to be key adjustment mediators for this age/SES group. In summary, although the present SPS training program strengthened children's problem-solving skills considerably, and to a lesser extent, aspects of their adjustment, it failed to show direct effects of SPS skill acquisition on adjustive gain for middle-SES kindergartners. That failure is vexing, because such a link undergirds primary prevention's rationale for training competencies in young children. The need to understand better the active agents in the mediation of adjustment-gain remains an important prerequisite for the robust development of future primary prevention work with young children.
Reference Notes 1. Krasnor, R.L., & Rubin, K.H. Preschoolers' verbal and behavioral solutions to social problems. Paper presented at Canadian Psychological Association, Ottawa, Ontario, June, 1978. 2. Shure, M.B., & Spivack, G. Cognitiveproblem-solving skills, adjustment and social class. Research and Evaluation Report. Philadelphia: Department of Mental Health Sciences, Hahnemann Medical College and Hospital, 1970. 3. Shure, M.B., & Spivack, G. Problem-solving capacity, social class and adjustment among nursery school children. Paper p r e s e n t e d at E a s t e r n Psychological Association, Atlantic City, New Jersey, 1970. 4. Shure, M.B., & Spivack, G. A mental health program for kindergarten children: Training script. Philadelphia: Department of Mental Health Sciences, Hahnemann Community Mental Health/Mental Retardation Center, 1974. 5. Shure, M.B., & Spivack, G. Preschool Interpersonal Problem Solving (PIPS] Test: Manual. Philadelphia: Department of Mental Health Sciences, Hahnemann Community Mental Health/Mental Retardation Center, 1974. 6. Spivack, G., & Levine, M. Self-regulation in acting-out and normal adolescents. Report M-4531, National Institute of Health, Washington, D.C., 1963.
References Albee, G.W. Mental health manpower trends. New York: Basic Books, 1959. Albee, G.W. The relation of conceptual models of disturbed behavior to institutional and manpower requirements. In F.N. Arhnoff, E.A. Rubenstein, & J.S. Speisman (Eds.), Manpower for mental health. Chicago: Aldine, 1969. Allen, G.J., Chinsky, J.M., Larcen, S.W., Lochman, J.E., & Selinger, H.V. Community psychology and the schools: A behaviorally oriented multilevel preventive approach. Hillsdale, N.J.: Lawrence Erlbaum Associates, 1976. Cowen, E.L. Baby-steps toward primary prevention. American Journal of Community Psychology, 1977, 5, 1-22. Cowen, E.L. The wooing of primary prevention. American Journal of Community Psychology, 1980, 8, 258-284. Dunn, L.M. Peabody Picture Vocabulary Test: Manual. Minnesota: American Guidance Service, Inc., 1965.
216
Journal of Primary Prevention
Elardo, P.T., & Caldwell, B.M. The effects of an experimental social development program on children in the middle childhood period. Psychology in the Schools, 1979, 16, 93-100. Gesten, E.L. A Health Resources Inventory: The development of a measure of the personal and social competence of primary-grade children. Journal of Consulting and Clinical Psychology, 1976, 44, 775-786. Gesten, E.L., Flores de Apodaca, R., Rains, M., Weissberg, R.P., & Cowen, E.L. Promoting peer related social competence in school. In M.W. Kent & J.E. Rolf (Eds.), The primary prevention of psychopathology. Volume 3: Promoting social competence and coping in children. Hanover, N.H.: University Press of New England, 1979. Griggs, J.W., & Bonney, M.E. Relationship between "causal" orientation and acceptance of others, "self-ideal self-congruence," and mental health changes for fourth and fifth grade children. Journal of Educational Research, 1970, 63, 471-477. Jahoda, M. The meaning of psychological health. Social Casework, 1953, 34, 349-354. Lorion, R.P., Cowen, E.L., & Caldwell, R.A. Normative data and parametric analyses of school maladjustment. American Journal of Community Psychology, 1975, 3, 291-301. Muuss, R.E. The effects of a one and two year causal learning program. Journal of Personality, 1960, 28, 479-491. Ojemann, R.H. Investigations on the effects of teacher understanding and appreciation of behavior dynamics. In G. Caplan (Ed.), Prevention of mental disorders in children. New York: Basic Books, 1961. Ojemann, R.H. Incorporating psychological concepts in the school curriculum. Journal of School Psychology, 1967, 5, 195-204. Shure, M.B., & Spivack, G. Problem solving techniques in childrearing. San Francisco: Jossey-Bass, 1978. Spivack, G., Platt, J.J., & Shure, M.B. The problem-solving approach to adjustment. San Francisco: Jossey-Bass, 1976. Spivack, G., & Shure, M.B. Social adjustment of young children: A cognitive approach to solving real life problems. San Francisco: Jossey-Bass, 1974. Weissberg, R.P., Gesten, E.L., Carnrike, C.L., Toro, P.A., Rapkin, B.D., Davidson, E., & Cowen, E.L. Social problem solving skills training: A competence-building intervention with 2nd-4th grade children, American Journal of Community Psychology, 1981, 9, 411-424. Weissberg, R.P., Gesten, E.L., Rapkin, B.D., Cowen, E.L., Davidson, E., Flores de Apodaca, R., & McKim, B. Evaluation of social-problem-solving training for suburban and inner-city third-grade children. Journal of Consulting and Clinical Psychology, 1981, 49, 251-261.
