Joumal of Advanced Nursing, 1990, 15, 316-323
Programme evaluation in nursing education: the state of the art Joan E Watson RN PhD Associate Professor and Director, Nursing Administration and Nursing Education Graduate Programmes
and Debra Herbener RN MN Doctoral Student Nurstng Educatton, School of Nurstng, Untverstty of Pittsburgh, Pittsburgh, USA
Accepted for publication i May 1989
WATSON J E & HERBENER D (1990) Journal of Advanced Nursmg 15, 316-323 Programme evaluation in nursing education: the state of the art Programme evaluation is a complex but integral component of a nursing education programme It is an ongoing process of collecting and describing data which provides the basis for decision making Specifically, evaluative data can be used to prepare for accreditation visits, account for budgetary expenditures, answer requests for information, develop faculty and staff, and examine the planned and actual effects of the programme withm the community and make changes accordmgly There are several programme evaluation models available to guide the evaluation process, but no single model is best and nurse educators must consider a variety of vanables Ideally, the selection of a model should be based on the purpose of the evaluation, programme needs, matenal and spatial resources, and personnel time, as well as the needs and desires of key mterest groups Several considerations are also required before the model is implemented These mclude determming specific pnonties for evaluation since all aspects of a programme (conceptual framework, philosophy, programme goals, student charactenstics, graduates' performance, faculty and adnunistrative expertise, as well as adequacy of resources) usually cannot be evaluated simultaneously, how the evaluation should proceed, time frames for specific evaluation projects, and personnel responsible for the evaluative activities
INTRODUCTION Designing systematic evaluations of educational programmes m schools of nursmg can be a difficult, timeconsuming, contentious, unrewarded process Yet there is little doubt in the literature of the programmatic value of a well-designed, systematic, evaluation for our nursmg education programmes The purpose of this article is to provide nurse educators and nurse educational administrators Correspondence Joan £ Watson Nursing Administration and Nursmg Education Graduate Programmes, Untoersity of Pittsburgh 426 Victoria Bmlding, 3500 Victoria Street Pittsburgh PA 15261 USA
316
With access to current concepts, pnnciples, and issues in nursing education evaluahon, including bnef descnptions of common evaluation models used m nursmg educahon DESCRIPTION, PURPOSE, PROCESS, OUTCOME Programme evaluahon mvolves a complex process of data collection and analysis m order to assign values to vanous programme components leading to deasions about the programme Posavac & Carey (1980) descnbed programme evaluation as a collechon of methods, skills and
Programme evaluation
sensitivities necessary to determine whether a human service IS needed and likely to be used, whether the service is offered as planned, and whether the human service actually does help people in need Applied to nursing education, programme evaluation provides a means by which the feasibility of a new programme can be determined, and the existence of a current programme can be justified, based on the extent to which the programmes effectively and efficiently meet the identified human service need of nursing ccire In general, the purposes of programme evaluation are to diagnose problems, weaknesses and strengths, test new and different approaches for accomplishing and advancing the school's philosophy, obiectives, and conceptual framework, and improve the operation of all aspects of the school (Poteet & PoUak 1986) More speafically, programme evaluation in nursmg education can help faculty and administrators account for scarce fiscal resouces, make administrative and cumcular deasions, appraise faculty and staff development needs, examine both mtended and unintended effects of their nursing programme within the community, and provide a mechanism to assure fulfilment of accreditation requirements The process of programme evaluation can involve several different activities and groups of people, depending upon the purpose of the evaluation Generally, collection of data IS governed by who or what will be evaluated, and when and how the evaluation will be accomplished Staropoli & Waltz (1978) delineated five key questions to answer m designing a comprehensive evaluation (1) Who will be involved in the evaluation? (2) What are the purposes of conducting the evaluation? (3) What is to be evaluated including cumcuium, objectives, faculty, students? (4) How IS the evaluation to proceed? (5) When should the evaluation occur? In order for the evaluation to be comprehensive, data collection should be ongomg, and evaluations can be both formative and summative A formative appraisal provides rationale for modifying the programme in some way while it is still m progress A summative evaluation, although utilizing data gathered continuously, emphasizes an end-of-programme, or summary appraisal of the strong and weak points of the programme This type of evaluation does not serve to modify existing programmes, but might influence future programmes Both evaluation processes assist in determining if the programme was implemented as designed and if it is servmg the persons for whom it was intended The outccMnes of programme evaluation provide a basis for decision-making that may justify maintaining certain components of the programme as they are, or suggest modificaUons or reorgaiuzation Often the assessment of
the outcome achieved by the p>eople involved in the programme becomes a major focus of the evaluation Outcome evaluation involves such questions as 'Are the graduates of this programme serving the target population as planned?,' or How do these graduates compare with graduates of previous years or other programmes in terms of nursing knowledge and practice?' Terminal competencies will be discussed m greater detail under student evaluation C O M P O N E N T S OF A NURSING E D U C A T I O N P R O G R A M M E T O BE EVALUATED The components of a nursing education programme that should be evaluated include the conceptual framework, philosophy, programme goals, the administration, cumcuium, faculty, outcomes (students and graduates), and ancillary resources Specific factors to consider for evaluation of each of these components are discussed below Conceptual framework The conceptual framework is the conceptualization and articulation of concepts, facts, propositions, postulates, theories, phenomena and vanables relevant to a specific nursing educational system (Bevis 1982) It provides the foundation for the nursing cumcuium, and as such, is perhaps just as important as the cumcuium itself Fawcett (1984) descnbed a process for conceptual framework analysis and evaluation which was successfully implemented by Winter et al (1987) Fawcett's process involves analysing the following aspects the histoncal evolution of the framework, the approach to the development of nursing knowledge as exemplified by the framework, the explication and relationships of nursing's four metaparadigm concepts (person, environment, health, and nursing), the areas of concem that are identified by the framework, and the source of these concems Evaluative questions involve an assessment of the clanty of underlying assumptions, completeness of descnphons of the concepts of person, environment, health, and nursing, the expliat nature of the relationships of the four concepts, the congruency, significance, and utility of the conceptual fi-amework, and the potential of the framework to yield empincally testable hypotheses in order to contnbute to nursing knowledge Philosophy According to Bevis (1982) the philosophy provides the picture window through which the world is viewed It is a 317
JE Watson andD Herbemr
speculation about the nature and value of things, and provides all interested parties with a clear and conase summary of the pnonties and commitments of the nursing programme As such, the philosophy should reflect the institutional philosophy, and serve as a guide for the development of the other components of the programme Evaluation of a nursing school's philosophical statement should first mdude an assessment of its eongrueney with the philosophy of the institution of whieh it is a part If the statements refleet divergent viewpoints, the sehool's philosophy should be revised Seeondly, the philosophy should help identify the purpose of the nursing programme Thirdly, it should be detailed and elear enough to guide the eonstruetion of a eoneeptual framework, and a system for aehieving the idenhfied purpose Lastly, the philosophy should represent the eonsensus of the faeulty who work from it, smee disagreements on fundamental viewpoints eould interfere with the proper funetiorung of other eomponents of the programme
Programme goals Programme goal statements, as they anse from the philosophy and eoneeptual framework, should also be evaluated Like the philosophy, programme goals should be eonsistent with the stated goals of the institution, and refleet the views of admmistrators, faeulty and students Aeeordmg to Line (1987), mcongruence of goal perceptions among these groups can lead to lnaeased student attntion rates, mefifective leadership, cind increased faculty turnover Programme goals should be dearly stated, and measurable, so that everyone understands them and their aehievement ean be determined A nursmg programme's goals should help justify the existenee of the programme withm the eoUegiate setting and the eommunity Coals should be revised penodieally to refleet ehanges in philosophy, eoneeptual frameworks, and trends m nursmg
EVALUATION OF ADMINISTRATION AND OPERATIONS An important eomponent of any programme evaluation m nursmg edueation is the evaluation of the admimsfration and operations of the programme The questions asked m regard to this phase of evaluation mdude (1) Do faeulty, admmistrators and students partieipate m the govemanee of the parent institution in aeeordance with the bylaws of the parent institution? (2) Does the organizational structure of the nursmg programme promote effective fimctionmg 318
and foster the attainment of programme goals7 (3) Is the programme administered by a nurse educator who holds a minimum of a bacealaureate m nursmg and an eamed doctoral degree and has expenence m baccalaureate or higher degree programmes m nursing? (4) Does the administrator of the nursmg programmes, with mstitutional consultation and nursing faculty input, have the responsibihty and authonty for plarmmg and allocating programme resourees? (National League for Nursmg 1983)
CURRICULUM EVALUATION The eumeulum is defined by Bevis (1982) as the totality of leammg aetivities that are designed to aehieve speafie edueational goals Evaluation of nursmg eumeula has beeome a major eoneem of nurse edueators in reeent years It is dosely linked to eumeulum development and therefore ean mvolve a proeess of eontinuous ehange The interest in improving nursing eumeula through evaluation is demonstrated by many authors (Anderson 1985, Clark et al 1983, Fields 1984, Horan et al, 1984, Jenkms 1986, Litwaek et al 1985, O'Neill 1986, Poteet & Hodges 1987, Stewart & Hluchyj 1987, Wakim 1983) The cumculum should reflect the philosophy, conceptual framework, and programme goals of the school and institution Cumeulum evaluation also mvolves an assessment of nursing eontent taught in eaeh eourse, eourse objeetives, teaehmg sfrategies, eourse evaluation methods, and the relationship of non-nursmg eourses to the overall plan of study Speafie items to eonsider m relation to course eontent melude the seleetion and organization of essential eontent from simple to complex, the relationship of this content to course objectives, and whether there are omissions or duplications of essential content, or mclusion of non-essential content Course objectives should descnbe expected behaviours and reflect the level and nature of eontent, whether eognitive, affeetive, or psyehomotor They should also be assessed for elanty, appropnateness to eontent, and the ease by which their achievement by students can be measured Teachmg strategies should relate to eourse eontent and objeetives, and allow for aehievement of the objeetives Course evaluationmethods mvolve both a determination by the faculty of whether students met the course objectives and an appraisal by the students regardmg the value of the eourse to them Evaluation of non-nursmg eourses should descnbe the extent to which they are used as a foundation for nursmg courses, or the extent to which they duplicate content taught m the nursing programme Revisions m the cumculum can be niade accordmgly
Programme evaluation
FACULTY EVALUATION Faculty evaluation is based on measunng performance in the areas of teachmg, research and service, and as such has been discussed, debated, maligned, and espoused by faculty, administrators, and students for some time (Seldin 1984) The necessity of faculty evaluahon is seldom queshoned However, the format and evidence used for the evaluahon are subjects of frequent discussion and debate Inshtutional and departmental philosophies can help speafy the pnonties and emphases that are assigned to these responsibibhes, which should in tum be reflected in the evaluative cntena Faculty evaluation should be based upon a comprehensive job descnption Evaluative data can be obtained from a number of sources, the most common of which include the faculty selfevaluahons, and external evaluations by administrators, peers and students Each data source cem provide specific information which will contnbute to a thorough evaluation Self-evaluation has traditionally provided the means by which the faculty can update their portfolio in terms of significant accomplishments in the areas of research, teachmg and services Accordmg to Hulsmeyer & Bowling (1986), faculty seldom volunteer their weaknesses m classroom teaching smce negative information could jeopardize their position Therefore, it is important to require faculty to list strengths and areas for improvement m their selfevaluations Also, supplementary, objective, evaluahon sources are cntical to comprehensive faculty evaluahon Admmistrative evaluahon of faculty generally mvolves synthesizmg data from vanous sources, and then reviewing results with faculty members, compared with predetermmed standards of performance These standards usually are tied to academic rank Somehmes administrative evaluation mvolves first hand observahon of classroom teachmg This IS usually done by the invitation of faculty who are mterested in formative input regarding their teachmg styles and strategies If not done on a voluntary basis, an administrator's presence m the classroom could alter normal interaction, resultmg m a biased evaluation, and potenhal lnfnngement of the faculty's academic freedom Peer evaluation is a somewhat controversial measure of faculty evaluahon, and often it is omitted from a comprehensive evaluahon of the faculty Mutual backscratchmg, fnendship, and populanty are seen as vanables that might skew peer evaluahon m a posihve direction On the other hand, mutual distrust and different views of effective teachmg may negahvely bias peer evaluahon Hulsmeyer & Bowling (1986) stated that perhaps the most significant cnhasm of peer evaluation is that it fosters a negahve and compehhve atmosphere among nursing faculty Peer
evaluahon is also time-consummg for faculty, which is particularly important smce faculty rewards are mcreasmgly based on scholarly achvity Combined with feedback from students and admmishators, however, Hulsmeyer & Bowling (1986) mamtamed that peer evaluahon can provide a balanced view of one's pertormance, and they developed a tool for such a purpose Areas m which peer evaluation may be helpful mclude evaluating professional competencies, appropnateness of matenals, effechveness of instructional methods, and use of speafic objectives m terms of scope, sequence and placement withm the cumculum Because ofthe sensitive nature of reviewing peers jmd the many factors that affect the validity of responses, the data obtamed from peer assessments should be used exclusively for faculty development rather than as a part of the formalized evaluahon system (Litwackeffl/ 1985, Seldin 1984) Student evaluations of faculty can provide important evaluative information regarding course assignments, textbooks, faimess of evaluation methods, mterest m the students, and interest m the subject matter They can however produce biased results since the evaluations may be based on such factors as expected course grades and personalities rather than actual teachmg effectiveness Students may also feel that their own grades will be affected by the way they evaluate faculty, or that the evaluation process has no real effect on producing positive change Another factor is that many students do not know how to evaluate faculty, or they are asked to evaluate faculty regarding matters on which they have no knowledge For these reasons, it is often more beneficial to consider student evaluations over a penod of time m order to identify consistent problems and pattems, rather than making significant and costly changes based on one set of student evaluahons However, student ratings have been shown to be a valid and reliable component of the measurement of teaching effectiveness when they were properly obtamed (French-Lazovik 1974, Lazovik 1979) A system of computenzed student evaluation of faculty was developed at the University of Pittsburgh The faculty member has a choice of three tools that measure classroom, dmical teachmg, and professional behaviours Results are mailed confidentially to faculty and identify specific areas of strengths and weaknesses Faculty may share these evaluations with appropnate administrators when bemg considered for retention, ment increase, promotion and tenure
EVALUATION OF OUTCOMES Evaluation of outcomes m nursing educahon programmes includes evaluation of students and graduates 319
/ £ Watson and D Herbener
Evaluation of students Students traditionally are evaluated throughout their academic careers so that their learning for each course of study can be assessed Using such methods as wntten and oral assignments, paper and pencil exammations, and clinical performance appraisals, values are assigned that reflect the student's achievement m a given area In addition to reflecting a particular student's achievement, such evaluative measures also reflect programme effectiveness m particular areas If collective deficiencies exist, for example m students' knowledge of pharmacology mathematics, this may pmpoint a weakness in the foundation courses, or m the nursing cumcuium itself Perhaps essential content is missing or course objectives are not reflected m testmg instruments Other student evaluation methods include the use of standardized tests, such as NLN examirwtions, although these are more summative m nature and test general nursmg knowledge rather than knowlege speafic to particular course objectives
Evaluation of graduates Graduates should be requested to evaluate selected aspects of the programme since they can identify strengths and weaknesses from the perspective of practitioner, rather than student Data collected from graduates should focus upon the degree to which termmal competences were achieved, and the ease with which the transition from student to practitioner was made Programme evaluation instruments should be sent to alumni within 3 to 6 months after graduation, to allow for a penod of onentation to their new positions Employers should also be asked to evaluate graduates' professional fjerformance, but these evaluations should be reviewed over tune so that general trends of strengths and weaknesses as well as unrealistic expectations of graduates can be identified Knowles et al (1985) developed instruments for programme evaluation by alumni, and for evaluation of alumni by their supervisors, after finding httle lnfoimation on such instruments in the nursmg hterature Another source of data for evaluation of programme outcomes is the consianers of nursing services Information from patients or clients regarding the effiaency and quality of nursmg services provided by speafic graduates is seldom obtained by schools of nursmg, perhaps because of logistical problems in data coUection However, this untapped reservoir of data could prove very informative regarding client satisfaction with nursmg care provided by graduates 320
Despite many of the drawbacks of outcome evaluation, it IS an essential component for comprehensive programme assessment If done properly, all of the above aspects of outcome evaluation can contnbute to form a mosaic of the total programme
EVALUATING ANCILLARY RESOURCES This aspect of evaluation should address the adequacy of services which support the o|>erations of the programme, such as hbranes, media and computer laboratones, and college and clinical laboratones Questions that should be addressed in this evaluation include Do the libranes have a sufficient number of nursmg books and journals for the number of nursing students they serve? Are the journals representative of nursing hterature? Are there adequate facilities for mdependent study, autotutonal, and computer assisted leanung? Does the college laboratory have sufftaent equipment and supplies for practice? How systematic IS the college laboratory's use? Are all of these faahties adequately strfed and open dunng hours convenient to a majonty of students? Are qualified faculty available to assist students in the college laboratory? How can clinical faahties accommodate the number of students m the programme, and do they provide adequate leammg opportunities? Are contracts with clinical faahties negotiated m such a way that leammg opportunities for students are maximized, le are students allowed to administer medications, can they use the faahties' libranes, do they have access to patients' medical records before their clinical expenences? This aspect of evaluation represents a major component of comprehensive programme evaluation plans
RESPONSIBILITY FOR EVALUATION Cnhcal to the success of the programme evaluation is the deasion of who will be responsible for the evaluation Nursing education evaluation can be conducted by intemal faculty or extemal evaluators Evaluation is most relevant when it builds upon and involves the people who are directly and mdirectly affected by its results (Staropoli & Waltz 1978) Nursmg faculty should be highly involved m evaluatmg their programmes because it is part of their academic responsibility, and faculty are most knowledgeable of and invested m the success of their own programmes Frequently, faculty members do not feel that they have the expertise required to implement a systematic programme evaluation When this is the case, rather than employ an extemal evaluator, an evaluahon consultant can
Programme evaluation
assist the faculty m conduchng their own evaluation It is useful, however, to designate one faculty member who will have primary responsibility for the evaluation process, m order to ensure that breakdowns m the total evaluation process and unnecessary duplication m effort do not occur (Waltz & Bond 1985) The designation of t b s evaluation leader should be made early m the evaluation process In addihon to some expertise in programme evaluation, this individual should possess the personal charactenshcs and abilities of cnhcal thmkmg, objective analysis, leadership, and communicahon skills If evaluation is to accomplish its purposes, those who collect and analyse evaluative data should be mvolved m the deasion making that the evaluation precipitates (Staropoli & Waltz 1978) The key audiences for the programme evaluation should be invited to share their opinions on the evaluation from the outset in order to insure co-operation and commitment throughout the process (Sohn 1987) Evaluation programmes that fail often do so because there was a distinction between those who conduct the evaluation and those who make the progranimahc deasions This failure can be avoided when each faculty member is held accountable for participating in the evaluahon effort
MODELS FOR EVALUATING NURSING EDUCATION PROGRAMMES Several programme and cumculum evaluahon models have been advanced in recent years Such strategies enhance both organizahon of the process and comprehensiveness of the evaluation Some of these models are bnefly descnbed below Comparative course evaluation Comparative course evaluahon is a process designed to identify articulation, duplication and omission of leaming opportunities between courses m a cumculum (Fields 1984) It involves systematic exammahon of course syllabi m order to identify omissions and duplications of content, and behavioural objectives between courses
upon one or more of the followmg (1) disaepanaes between programme design and actual programme operations, such as planned cumculum content, and actual content taught, (2) discrepancies between predicted and obtained programme outcomes, such as predicted and actual NCLEX scores, (3) discrepancies between student achievement and desired standards of competency, (4) goal disaepanaes, or differences m value judgements of different groups, regarding the purpose or effectiveness of the programme, mdudmg progrcimme administrators, faculty, students, potential employers, and consumers withm the community, and (5) discrepancies between interchangeable components within the cumculum, such as mulhple course offenngs of the same subject where different instructors may have different course emphases Goal free evaluation The focus of goal free evaluation is to determme the importance and value of fmal outcomes, not intenhons and design of the programme It is an approach proposed by Scnven (1972) as a means of ensunng that evaluators take mto account the actual effects and not just the intended effects of education and training programmes In this case, the evaluator makes speaal efforts not to be influenced by programme goals, which are relevant to programme planning and development but not to the programme evaluation (Sohn 1987) Stakes Countenance Model Accordmg to Stakes (1967) the two basic acts of evaluation are descnption and judgement, both of which are essential if educational programmes are to be understood In Stake's Countenance Model, descnphve data are classified as intents (programme goals) and observations Programme goals and observations are congruent if what was mtended actually happened Judgement involves deasions regeffdmg the ment or worth of an educational practice and has two basis (I) absolute standards as reflected by personal judgments, and (2) relative standards as reflected by charactenstics of alternate programmes The judgmg act itself is deadmg which set of standards to heed and whether to take admmistrahve action
Discrepancy evaluation model Discrepancy evaluation refers to the search for differences between two or more elements or vanables of an education/ trammg programme that, accordmg to logical, rational or stahshcal cntena, should be in agreement or correspondence (Anderson rf ai 1975) Most commonly the focus of programme evaluation usmg the discrepancy model is
Staropoli and Waltz Model The Staropoli and Waltz Model (Staropoli & Waltz 1978) IS a comprehensive approach to educational programme evaluahon that has been widely used m schools of nursmg It offers a prototype than can comprehensively incorporate 321
J E Watson and D Herbener
the philosophy, structure and processes m effect m most schools of nursmg This model is based on five major decisions that must be made by the evaluators These decisions represent a dynamic process m that they are continually evaluated and revised as necessary The Waltz evaluation plan considers the educahonal programme as an open system consisting of inputs, operations and outputs Induded in the plan are four distmct yet mtenelated levels of analysis school, programme, subprogramme, and course level Waltz has developed instruments for use with this evaluahon model that can serve as a framework for the instruments which will compnse the evaluation plan as it is individucilized to a school of nursing These instruments developed by Waltz make her model appealing since they deaease the amount of work which must be created from saatch by the school of nursing faculty Stufflebeam's CIPP Model Context, input, process and product (CIPP) compnse the four types of evaluation m Stufflebeam's model (Patton 1982) Context evaluation serves decision-making for the plannmg of an ongoing programme and therefore is formative or diagnostic in nature The end products of context evaluation are programme chjmges which presumably will result in smaller discrepancies between mtended and acutal outcomes (Anderson et al 1975) The purpose, goals, and demand for the programme withm the community fall under context evaluahon Input evaluation provides mformation about the means necessary and available to reach the programme goals Input evaluahon involves questions regarding the feasibility, availability, costs, and potential advantages and disadvantages of the vanous strategies proposed Process evaluahon provides feedback to administrators and partiapants of the programme which detects or predicts problems with the day-to-day functioning of the programme The more speafic the mput evaluahon, the easier the process evaluation Thefinalcomponent, product evaluahon, serves to measure the extent to which goals have been achieved, and is therefore summahve m nature This model was implemented by Clark et al (1983) SELECTING AN EVALUATION MODEL Selection of a model to assist m programme evaluation should be based on the purpose of the evaluahon, the programme needs, resources, and time It is prudent for nurse educators to review carefully the literature on programme evaluahon and the available evaluahon models 322
Once the model is selected, the evaluators must feel free to adapt the model to their own nursmg education programme Although models can provide a general framework, they are of little help m explaming the specifics of the actual evaluation process (Sohn 1987) Therefore, the individual responsible for the evaluation must have flexibility in the manner m which the model is implemented Evaluators should be encouraged by the school of nursmg administrators to modify the selected model to fit the nursmg education programme bemg evaluated CONCLUSION Although programme evaluahon can be time-consuming and expensive, it is necessary for development and nuuntenance of quahty nursing education programmes Escalating costs, shnnkmg appropnations, and the knowledge explosion have contnbuted to the need for examining educational programmes for efficiency and efficacy The cost of teaching nursmg students dmical skills is of particular concem smce the dmical faculty spend a great deal of time on a one-to-one basis with each student Students must also be taught more theory and psychomotor skills now than ever before, but often by the same or fewer number of faculty Administrators and faculty are accountable to consumers of nursing services, health care agenaes, students, and the institution as a whole to make sure the programme meets or exceeds acaeditation standards and that the programme is mabng the best use of scarce resources The scarcity of resouces not only involvesfisccilappropnations, but laboratory facilities as well With competition among several schools of nursing for access to the same clmical facilities, it is particularly important to utilize hospitals, nursmg homes, dimes and home health care agencies effectively so that leammg opportunities for students are maximized Bndgmg the gap between nursing education and nursmg practice is also faahtated m this way, smce programme evaluahon identifies problem areas that should be addressed There also is compehtion among schools of nursing for nursing students As a result many schools are offermg aeahve educahonal programmes m an attempt to attract students The demand and efficacy of such programmes, as well as traditional programmes, need to be conhnually evaluated An ongomg evaluahon process can help assure that mshtuhonal and programme goals are fulfilled, and establish the school's reputahon for a quality programme, which will result m attracting students Continued researdi on programme evaluahon is recommended The literature reveals a pauaty of studies
Programme evaluation Knowles L, Strozier V, Wilson J, Bodo T , Greene D & Sarver V (1985) Evaluation of a baccalaureate nursing program by alumni and of alumni by their supervisors Joumal of Nursing Education 24,261-264 Lazovik G (1979) The Report to the School of Nursing Committee on the Evaluation of Teaching (SONCET) on the Development of Student Evaluation of Teaching Scales Office for the Evaluahon of Teaching, Uruversity of Pittsburgh, Pittsburgh LincL (1987) Institutional goal analysis an approach to program evaluation Joumal of Nursmg Education 26,172-175 Litwack L, Line L & Bower D (1985) Evaluation m Nursmg Pnnaples and Practice Nahonal League for Nursing, New York Nursing schools in the future will face even greater pp 1-270 challenges for produchvity, with greater accountability for Nahonal League for Nursmg (1983) Cntena for the Evaluation of Baccalaureate and Higher Degree Programs m Nursing, 5th edn administrators and faculty A aedible, ongoing evaluation National League for Nursing, New York, pp 1-16 plan that is individualized to a particular programme is O'Neill E (1986) Comprehensive cumculum evaluation Joumal necessary in order to maintain our high standards of of Nursing Education 25, 37-39 nursmg education Patton M (1982) Practical Evaluation Sage, Beverly Hills, p 38 Posavac EJ & Carey RG (1980) Program Evaluation Methods and Case Studies Prentice-Hall, Englewood Cliffs, p 6 References PoteetG & Hodges L (1987) Strategies for ensunng a successful cumculum consultation Nurse Educator 12(5), 25—28 Anderson J (1985) Why nursing education needs curriculum Poteet G & PoUak C (1986) Increasing accountability through speaalists Nursing Outlook 3 3 , 96-97 program evaluation Nurse Educator 11(2), 4 1 - 4 7 Anderson S B , Ball S & Murphy R T (1975) Encyclopedia of Scnven M (1972) Pros and cons about goal-free evaluation Educational Evaluation Jossey-Bass, San Francisco Evaluation Comment 3,1—4 Bevis E (1982) Cumculum Building in Nurstng, 3rd edn C V Seldin P (1984) Changmg Practices in Faculty Evaluation JosseyMosby, St Louis, pp 26, 34, 35 Bass, Scin Fransisco, p 76 Clark T , Goodwin M , Manani M , Marshall M & Moore S (1983) Curriculum evaluation, an applicahon of Stufflebeam's Sohn K (1987) Program evaluation m nursmg Nurse Educator 12(2), 27-33 model in a baccalaureate school of nursing Joumal of Nursing Stakes R (1967) The countenance of educational evaluation Education 22, 54-58 Teachers College Record 68, 523-540 Fawcett J (1984) Analysis and Evaluation of Conceptual Models m Staropoli CJ & Waltz CF (1978) Developing and Evaluating Nursmg F A Davis, Philadelphia, pp 37-47 Educational Programs for Health Care Providers F A Davis, Fields M (1984) A model for comparahve course evaluation Philadelphia Joumal of Nursmg Education 2 3 , 76-78 French-Lazovik G (1974) Predictability of students' evaluahons Stewart J & Hluchyj T (1987) An approach to nursmg cumculum evaluation Nursing Outlook 35, 79-81 of college teachers from component ratings Joumal of Wakim J (1983) A workable m-depth cumculum evaluation plan Educational Psychology 66,373-385 Nurse Educator 8(3), 24-32 Horan M , Knight L, McAtee S & Weshick J (1984) A model for cumculum evaluahon Joumal of Nursing Education 2 3 , Waltz C & Bond A (1985) How can a program evaluation be comprehensive and yet cost efficient? Joumal of Nursing 319-322 Education 24, 258-261 Hulsmeyer B & Bowling A (1986) Evaluating colleagues' classWinter E, Bender A , Hertz J & Reider J (1987) Analyzing and room teaching effectiveness Nurse Educator 11(5), 19-23 evaluating a baccalaureate nursing cumculum framework Jenkins H (1986) Student participation in curriculum evaluation Nurse Educator 12(4), 10-13 Nurse Educator 11(5), 16-18
regarding the development and testing of quantitahve evaluahon tools m nursmg education Qualitahve studies are also recommended A phenomenological study exploring the meaning of a nursing educahon programme to key interest groups, for example, would provide perspectives that would be obscured in a quantitative approach that measures students' cogrutive achievement Both types of data can provide valuable insights regarding areas of strengths and weaknesses of a programme If used in complimentary fashion, both quantitahve and qualitative methods will contnbute to a comprehensive evaluation
323
Programme evaluation in nursing education: the state of the art Joan E Watson RN PhD Associate Professor and Director, Nursing Administration and Nursing Education Graduate Programmes
and Debra Herbener RN MN Doctoral Student Nurstng Educatton, School of Nurstng, Untverstty of Pittsburgh, Pittsburgh, USA
Accepted for publication i May 1989
WATSON J E & HERBENER D (1990) Journal of Advanced Nursmg 15, 316-323 Programme evaluation in nursing education: the state of the art Programme evaluation is a complex but integral component of a nursing education programme It is an ongoing process of collecting and describing data which provides the basis for decision making Specifically, evaluative data can be used to prepare for accreditation visits, account for budgetary expenditures, answer requests for information, develop faculty and staff, and examine the planned and actual effects of the programme withm the community and make changes accordmgly There are several programme evaluation models available to guide the evaluation process, but no single model is best and nurse educators must consider a variety of vanables Ideally, the selection of a model should be based on the purpose of the evaluation, programme needs, matenal and spatial resources, and personnel time, as well as the needs and desires of key mterest groups Several considerations are also required before the model is implemented These mclude determming specific pnonties for evaluation since all aspects of a programme (conceptual framework, philosophy, programme goals, student charactenstics, graduates' performance, faculty and adnunistrative expertise, as well as adequacy of resources) usually cannot be evaluated simultaneously, how the evaluation should proceed, time frames for specific evaluation projects, and personnel responsible for the evaluative activities
INTRODUCTION Designing systematic evaluations of educational programmes m schools of nursmg can be a difficult, timeconsuming, contentious, unrewarded process Yet there is little doubt in the literature of the programmatic value of a well-designed, systematic, evaluation for our nursmg education programmes The purpose of this article is to provide nurse educators and nurse educational administrators Correspondence Joan £ Watson Nursing Administration and Nursmg Education Graduate Programmes, Untoersity of Pittsburgh 426 Victoria Bmlding, 3500 Victoria Street Pittsburgh PA 15261 USA
316
With access to current concepts, pnnciples, and issues in nursing education evaluahon, including bnef descnptions of common evaluation models used m nursmg educahon DESCRIPTION, PURPOSE, PROCESS, OUTCOME Programme evaluahon mvolves a complex process of data collection and analysis m order to assign values to vanous programme components leading to deasions about the programme Posavac & Carey (1980) descnbed programme evaluation as a collechon of methods, skills and
Programme evaluation
sensitivities necessary to determine whether a human service IS needed and likely to be used, whether the service is offered as planned, and whether the human service actually does help people in need Applied to nursing education, programme evaluation provides a means by which the feasibility of a new programme can be determined, and the existence of a current programme can be justified, based on the extent to which the programmes effectively and efficiently meet the identified human service need of nursing ccire In general, the purposes of programme evaluation are to diagnose problems, weaknesses and strengths, test new and different approaches for accomplishing and advancing the school's philosophy, obiectives, and conceptual framework, and improve the operation of all aspects of the school (Poteet & PoUak 1986) More speafically, programme evaluation in nursmg education can help faculty and administrators account for scarce fiscal resouces, make administrative and cumcular deasions, appraise faculty and staff development needs, examine both mtended and unintended effects of their nursing programme within the community, and provide a mechanism to assure fulfilment of accreditation requirements The process of programme evaluation can involve several different activities and groups of people, depending upon the purpose of the evaluation Generally, collection of data IS governed by who or what will be evaluated, and when and how the evaluation will be accomplished Staropoli & Waltz (1978) delineated five key questions to answer m designing a comprehensive evaluation (1) Who will be involved in the evaluation? (2) What are the purposes of conducting the evaluation? (3) What is to be evaluated including cumcuium, objectives, faculty, students? (4) How IS the evaluation to proceed? (5) When should the evaluation occur? In order for the evaluation to be comprehensive, data collection should be ongomg, and evaluations can be both formative and summative A formative appraisal provides rationale for modifying the programme in some way while it is still m progress A summative evaluation, although utilizing data gathered continuously, emphasizes an end-of-programme, or summary appraisal of the strong and weak points of the programme This type of evaluation does not serve to modify existing programmes, but might influence future programmes Both evaluation processes assist in determining if the programme was implemented as designed and if it is servmg the persons for whom it was intended The outccMnes of programme evaluation provide a basis for decision-making that may justify maintaining certain components of the programme as they are, or suggest modificaUons or reorgaiuzation Often the assessment of
the outcome achieved by the p>eople involved in the programme becomes a major focus of the evaluation Outcome evaluation involves such questions as 'Are the graduates of this programme serving the target population as planned?,' or How do these graduates compare with graduates of previous years or other programmes in terms of nursing knowledge and practice?' Terminal competencies will be discussed m greater detail under student evaluation C O M P O N E N T S OF A NURSING E D U C A T I O N P R O G R A M M E T O BE EVALUATED The components of a nursing education programme that should be evaluated include the conceptual framework, philosophy, programme goals, the administration, cumcuium, faculty, outcomes (students and graduates), and ancillary resources Specific factors to consider for evaluation of each of these components are discussed below Conceptual framework The conceptual framework is the conceptualization and articulation of concepts, facts, propositions, postulates, theories, phenomena and vanables relevant to a specific nursing educational system (Bevis 1982) It provides the foundation for the nursing cumcuium, and as such, is perhaps just as important as the cumcuium itself Fawcett (1984) descnbed a process for conceptual framework analysis and evaluation which was successfully implemented by Winter et al (1987) Fawcett's process involves analysing the following aspects the histoncal evolution of the framework, the approach to the development of nursing knowledge as exemplified by the framework, the explication and relationships of nursing's four metaparadigm concepts (person, environment, health, and nursing), the areas of concem that are identified by the framework, and the source of these concems Evaluative questions involve an assessment of the clanty of underlying assumptions, completeness of descnphons of the concepts of person, environment, health, and nursing, the expliat nature of the relationships of the four concepts, the congruency, significance, and utility of the conceptual fi-amework, and the potential of the framework to yield empincally testable hypotheses in order to contnbute to nursing knowledge Philosophy According to Bevis (1982) the philosophy provides the picture window through which the world is viewed It is a 317
JE Watson andD Herbemr
speculation about the nature and value of things, and provides all interested parties with a clear and conase summary of the pnonties and commitments of the nursing programme As such, the philosophy should reflect the institutional philosophy, and serve as a guide for the development of the other components of the programme Evaluation of a nursing school's philosophical statement should first mdude an assessment of its eongrueney with the philosophy of the institution of whieh it is a part If the statements refleet divergent viewpoints, the sehool's philosophy should be revised Seeondly, the philosophy should help identify the purpose of the nursing programme Thirdly, it should be detailed and elear enough to guide the eonstruetion of a eoneeptual framework, and a system for aehieving the idenhfied purpose Lastly, the philosophy should represent the eonsensus of the faeulty who work from it, smee disagreements on fundamental viewpoints eould interfere with the proper funetiorung of other eomponents of the programme
Programme goals Programme goal statements, as they anse from the philosophy and eoneeptual framework, should also be evaluated Like the philosophy, programme goals should be eonsistent with the stated goals of the institution, and refleet the views of admmistrators, faeulty and students Aeeordmg to Line (1987), mcongruence of goal perceptions among these groups can lead to lnaeased student attntion rates, mefifective leadership, cind increased faculty turnover Programme goals should be dearly stated, and measurable, so that everyone understands them and their aehievement ean be determined A nursmg programme's goals should help justify the existenee of the programme withm the eoUegiate setting and the eommunity Coals should be revised penodieally to refleet ehanges in philosophy, eoneeptual frameworks, and trends m nursmg
EVALUATION OF ADMINISTRATION AND OPERATIONS An important eomponent of any programme evaluation m nursmg edueation is the evaluation of the admimsfration and operations of the programme The questions asked m regard to this phase of evaluation mdude (1) Do faeulty, admmistrators and students partieipate m the govemanee of the parent institution in aeeordance with the bylaws of the parent institution? (2) Does the organizational structure of the nursmg programme promote effective fimctionmg 318
and foster the attainment of programme goals7 (3) Is the programme administered by a nurse educator who holds a minimum of a bacealaureate m nursmg and an eamed doctoral degree and has expenence m baccalaureate or higher degree programmes m nursing? (4) Does the administrator of the nursmg programmes, with mstitutional consultation and nursing faculty input, have the responsibihty and authonty for plarmmg and allocating programme resourees? (National League for Nursmg 1983)
CURRICULUM EVALUATION The eumeulum is defined by Bevis (1982) as the totality of leammg aetivities that are designed to aehieve speafie edueational goals Evaluation of nursmg eumeula has beeome a major eoneem of nurse edueators in reeent years It is dosely linked to eumeulum development and therefore ean mvolve a proeess of eontinuous ehange The interest in improving nursing eumeula through evaluation is demonstrated by many authors (Anderson 1985, Clark et al 1983, Fields 1984, Horan et al, 1984, Jenkms 1986, Litwaek et al 1985, O'Neill 1986, Poteet & Hodges 1987, Stewart & Hluchyj 1987, Wakim 1983) The cumculum should reflect the philosophy, conceptual framework, and programme goals of the school and institution Cumeulum evaluation also mvolves an assessment of nursing eontent taught in eaeh eourse, eourse objeetives, teaehmg sfrategies, eourse evaluation methods, and the relationship of non-nursmg eourses to the overall plan of study Speafie items to eonsider m relation to course eontent melude the seleetion and organization of essential eontent from simple to complex, the relationship of this content to course objectives, and whether there are omissions or duplications of essential content, or mclusion of non-essential content Course objectives should descnbe expected behaviours and reflect the level and nature of eontent, whether eognitive, affeetive, or psyehomotor They should also be assessed for elanty, appropnateness to eontent, and the ease by which their achievement by students can be measured Teachmg strategies should relate to eourse eontent and objeetives, and allow for aehievement of the objeetives Course evaluationmethods mvolve both a determination by the faculty of whether students met the course objectives and an appraisal by the students regardmg the value of the eourse to them Evaluation of non-nursmg eourses should descnbe the extent to which they are used as a foundation for nursmg courses, or the extent to which they duplicate content taught m the nursing programme Revisions m the cumculum can be niade accordmgly
Programme evaluation
FACULTY EVALUATION Faculty evaluation is based on measunng performance in the areas of teachmg, research and service, and as such has been discussed, debated, maligned, and espoused by faculty, administrators, and students for some time (Seldin 1984) The necessity of faculty evaluahon is seldom queshoned However, the format and evidence used for the evaluahon are subjects of frequent discussion and debate Inshtutional and departmental philosophies can help speafy the pnonties and emphases that are assigned to these responsibibhes, which should in tum be reflected in the evaluative cntena Faculty evaluation should be based upon a comprehensive job descnption Evaluative data can be obtained from a number of sources, the most common of which include the faculty selfevaluahons, and external evaluations by administrators, peers and students Each data source cem provide specific information which will contnbute to a thorough evaluation Self-evaluation has traditionally provided the means by which the faculty can update their portfolio in terms of significant accomplishments in the areas of research, teachmg and services Accordmg to Hulsmeyer & Bowling (1986), faculty seldom volunteer their weaknesses m classroom teaching smce negative information could jeopardize their position Therefore, it is important to require faculty to list strengths and areas for improvement m their selfevaluations Also, supplementary, objective, evaluahon sources are cntical to comprehensive faculty evaluahon Admmistrative evaluahon of faculty generally mvolves synthesizmg data from vanous sources, and then reviewing results with faculty members, compared with predetermmed standards of performance These standards usually are tied to academic rank Somehmes administrative evaluation mvolves first hand observahon of classroom teachmg This IS usually done by the invitation of faculty who are mterested in formative input regarding their teachmg styles and strategies If not done on a voluntary basis, an administrator's presence m the classroom could alter normal interaction, resultmg m a biased evaluation, and potenhal lnfnngement of the faculty's academic freedom Peer evaluation is a somewhat controversial measure of faculty evaluahon, and often it is omitted from a comprehensive evaluahon of the faculty Mutual backscratchmg, fnendship, and populanty are seen as vanables that might skew peer evaluahon m a posihve direction On the other hand, mutual distrust and different views of effective teachmg may negahvely bias peer evaluahon Hulsmeyer & Bowling (1986) stated that perhaps the most significant cnhasm of peer evaluation is that it fosters a negahve and compehhve atmosphere among nursing faculty Peer
evaluahon is also time-consummg for faculty, which is particularly important smce faculty rewards are mcreasmgly based on scholarly achvity Combined with feedback from students and admmishators, however, Hulsmeyer & Bowling (1986) mamtamed that peer evaluahon can provide a balanced view of one's pertormance, and they developed a tool for such a purpose Areas m which peer evaluation may be helpful mclude evaluating professional competencies, appropnateness of matenals, effechveness of instructional methods, and use of speafic objectives m terms of scope, sequence and placement withm the cumculum Because ofthe sensitive nature of reviewing peers jmd the many factors that affect the validity of responses, the data obtamed from peer assessments should be used exclusively for faculty development rather than as a part of the formalized evaluahon system (Litwackeffl/ 1985, Seldin 1984) Student evaluations of faculty can provide important evaluative information regarding course assignments, textbooks, faimess of evaluation methods, mterest m the students, and interest m the subject matter They can however produce biased results since the evaluations may be based on such factors as expected course grades and personalities rather than actual teachmg effectiveness Students may also feel that their own grades will be affected by the way they evaluate faculty, or that the evaluation process has no real effect on producing positive change Another factor is that many students do not know how to evaluate faculty, or they are asked to evaluate faculty regarding matters on which they have no knowledge For these reasons, it is often more beneficial to consider student evaluations over a penod of time m order to identify consistent problems and pattems, rather than making significant and costly changes based on one set of student evaluahons However, student ratings have been shown to be a valid and reliable component of the measurement of teaching effectiveness when they were properly obtamed (French-Lazovik 1974, Lazovik 1979) A system of computenzed student evaluation of faculty was developed at the University of Pittsburgh The faculty member has a choice of three tools that measure classroom, dmical teachmg, and professional behaviours Results are mailed confidentially to faculty and identify specific areas of strengths and weaknesses Faculty may share these evaluations with appropnate administrators when bemg considered for retention, ment increase, promotion and tenure
EVALUATION OF OUTCOMES Evaluation of outcomes m nursing educahon programmes includes evaluation of students and graduates 319
/ £ Watson and D Herbener
Evaluation of students Students traditionally are evaluated throughout their academic careers so that their learning for each course of study can be assessed Using such methods as wntten and oral assignments, paper and pencil exammations, and clinical performance appraisals, values are assigned that reflect the student's achievement m a given area In addition to reflecting a particular student's achievement, such evaluative measures also reflect programme effectiveness m particular areas If collective deficiencies exist, for example m students' knowledge of pharmacology mathematics, this may pmpoint a weakness in the foundation courses, or m the nursing cumcuium itself Perhaps essential content is missing or course objectives are not reflected m testmg instruments Other student evaluation methods include the use of standardized tests, such as NLN examirwtions, although these are more summative m nature and test general nursmg knowledge rather than knowlege speafic to particular course objectives
Evaluation of graduates Graduates should be requested to evaluate selected aspects of the programme since they can identify strengths and weaknesses from the perspective of practitioner, rather than student Data collected from graduates should focus upon the degree to which termmal competences were achieved, and the ease with which the transition from student to practitioner was made Programme evaluation instruments should be sent to alumni within 3 to 6 months after graduation, to allow for a penod of onentation to their new positions Employers should also be asked to evaluate graduates' professional fjerformance, but these evaluations should be reviewed over tune so that general trends of strengths and weaknesses as well as unrealistic expectations of graduates can be identified Knowles et al (1985) developed instruments for programme evaluation by alumni, and for evaluation of alumni by their supervisors, after finding httle lnfoimation on such instruments in the nursmg hterature Another source of data for evaluation of programme outcomes is the consianers of nursing services Information from patients or clients regarding the effiaency and quality of nursmg services provided by speafic graduates is seldom obtained by schools of nursmg, perhaps because of logistical problems in data coUection However, this untapped reservoir of data could prove very informative regarding client satisfaction with nursmg care provided by graduates 320
Despite many of the drawbacks of outcome evaluation, it IS an essential component for comprehensive programme assessment If done properly, all of the above aspects of outcome evaluation can contnbute to form a mosaic of the total programme
EVALUATING ANCILLARY RESOURCES This aspect of evaluation should address the adequacy of services which support the o|>erations of the programme, such as hbranes, media and computer laboratones, and college and clinical laboratones Questions that should be addressed in this evaluation include Do the libranes have a sufficient number of nursmg books and journals for the number of nursing students they serve? Are the journals representative of nursing hterature? Are there adequate facilities for mdependent study, autotutonal, and computer assisted leanung? Does the college laboratory have sufftaent equipment and supplies for practice? How systematic IS the college laboratory's use? Are all of these faahties adequately strfed and open dunng hours convenient to a majonty of students? Are qualified faculty available to assist students in the college laboratory? How can clinical faahties accommodate the number of students m the programme, and do they provide adequate leammg opportunities? Are contracts with clinical faahties negotiated m such a way that leammg opportunities for students are maximized, le are students allowed to administer medications, can they use the faahties' libranes, do they have access to patients' medical records before their clinical expenences? This aspect of evaluation represents a major component of comprehensive programme evaluation plans
RESPONSIBILITY FOR EVALUATION Cnhcal to the success of the programme evaluation is the deasion of who will be responsible for the evaluation Nursing education evaluation can be conducted by intemal faculty or extemal evaluators Evaluation is most relevant when it builds upon and involves the people who are directly and mdirectly affected by its results (Staropoli & Waltz 1978) Nursmg faculty should be highly involved m evaluatmg their programmes because it is part of their academic responsibility, and faculty are most knowledgeable of and invested m the success of their own programmes Frequently, faculty members do not feel that they have the expertise required to implement a systematic programme evaluation When this is the case, rather than employ an extemal evaluator, an evaluahon consultant can
Programme evaluation
assist the faculty m conduchng their own evaluation It is useful, however, to designate one faculty member who will have primary responsibility for the evaluation process, m order to ensure that breakdowns m the total evaluation process and unnecessary duplication m effort do not occur (Waltz & Bond 1985) The designation of t b s evaluation leader should be made early m the evaluation process In addihon to some expertise in programme evaluation, this individual should possess the personal charactenshcs and abilities of cnhcal thmkmg, objective analysis, leadership, and communicahon skills If evaluation is to accomplish its purposes, those who collect and analyse evaluative data should be mvolved m the deasion making that the evaluation precipitates (Staropoli & Waltz 1978) The key audiences for the programme evaluation should be invited to share their opinions on the evaluation from the outset in order to insure co-operation and commitment throughout the process (Sohn 1987) Evaluation programmes that fail often do so because there was a distinction between those who conduct the evaluation and those who make the progranimahc deasions This failure can be avoided when each faculty member is held accountable for participating in the evaluahon effort
MODELS FOR EVALUATING NURSING EDUCATION PROGRAMMES Several programme and cumculum evaluahon models have been advanced in recent years Such strategies enhance both organizahon of the process and comprehensiveness of the evaluation Some of these models are bnefly descnbed below Comparative course evaluation Comparative course evaluahon is a process designed to identify articulation, duplication and omission of leaming opportunities between courses m a cumculum (Fields 1984) It involves systematic exammahon of course syllabi m order to identify omissions and duplications of content, and behavioural objectives between courses
upon one or more of the followmg (1) disaepanaes between programme design and actual programme operations, such as planned cumculum content, and actual content taught, (2) discrepancies between predicted and obtained programme outcomes, such as predicted and actual NCLEX scores, (3) discrepancies between student achievement and desired standards of competency, (4) goal disaepanaes, or differences m value judgements of different groups, regarding the purpose or effectiveness of the programme, mdudmg progrcimme administrators, faculty, students, potential employers, and consumers withm the community, and (5) discrepancies between interchangeable components within the cumculum, such as mulhple course offenngs of the same subject where different instructors may have different course emphases Goal free evaluation The focus of goal free evaluation is to determme the importance and value of fmal outcomes, not intenhons and design of the programme It is an approach proposed by Scnven (1972) as a means of ensunng that evaluators take mto account the actual effects and not just the intended effects of education and training programmes In this case, the evaluator makes speaal efforts not to be influenced by programme goals, which are relevant to programme planning and development but not to the programme evaluation (Sohn 1987) Stakes Countenance Model Accordmg to Stakes (1967) the two basic acts of evaluation are descnption and judgement, both of which are essential if educational programmes are to be understood In Stake's Countenance Model, descnphve data are classified as intents (programme goals) and observations Programme goals and observations are congruent if what was mtended actually happened Judgement involves deasions regeffdmg the ment or worth of an educational practice and has two basis (I) absolute standards as reflected by personal judgments, and (2) relative standards as reflected by charactenstics of alternate programmes The judgmg act itself is deadmg which set of standards to heed and whether to take admmistrahve action
Discrepancy evaluation model Discrepancy evaluation refers to the search for differences between two or more elements or vanables of an education/ trammg programme that, accordmg to logical, rational or stahshcal cntena, should be in agreement or correspondence (Anderson rf ai 1975) Most commonly the focus of programme evaluation usmg the discrepancy model is
Staropoli and Waltz Model The Staropoli and Waltz Model (Staropoli & Waltz 1978) IS a comprehensive approach to educational programme evaluahon that has been widely used m schools of nursmg It offers a prototype than can comprehensively incorporate 321
J E Watson and D Herbener
the philosophy, structure and processes m effect m most schools of nursmg This model is based on five major decisions that must be made by the evaluators These decisions represent a dynamic process m that they are continually evaluated and revised as necessary The Waltz evaluation plan considers the educahonal programme as an open system consisting of inputs, operations and outputs Induded in the plan are four distmct yet mtenelated levels of analysis school, programme, subprogramme, and course level Waltz has developed instruments for use with this evaluahon model that can serve as a framework for the instruments which will compnse the evaluation plan as it is individucilized to a school of nursing These instruments developed by Waltz make her model appealing since they deaease the amount of work which must be created from saatch by the school of nursing faculty Stufflebeam's CIPP Model Context, input, process and product (CIPP) compnse the four types of evaluation m Stufflebeam's model (Patton 1982) Context evaluation serves decision-making for the plannmg of an ongoing programme and therefore is formative or diagnostic in nature The end products of context evaluation are programme chjmges which presumably will result in smaller discrepancies between mtended and acutal outcomes (Anderson et al 1975) The purpose, goals, and demand for the programme withm the community fall under context evaluahon Input evaluation provides mformation about the means necessary and available to reach the programme goals Input evaluahon involves questions regarding the feasibility, availability, costs, and potential advantages and disadvantages of the vanous strategies proposed Process evaluahon provides feedback to administrators and partiapants of the programme which detects or predicts problems with the day-to-day functioning of the programme The more speafic the mput evaluahon, the easier the process evaluation Thefinalcomponent, product evaluahon, serves to measure the extent to which goals have been achieved, and is therefore summahve m nature This model was implemented by Clark et al (1983) SELECTING AN EVALUATION MODEL Selection of a model to assist m programme evaluation should be based on the purpose of the evaluahon, the programme needs, resources, and time It is prudent for nurse educators to review carefully the literature on programme evaluahon and the available evaluahon models 322
Once the model is selected, the evaluators must feel free to adapt the model to their own nursmg education programme Although models can provide a general framework, they are of little help m explaming the specifics of the actual evaluation process (Sohn 1987) Therefore, the individual responsible for the evaluation must have flexibility in the manner m which the model is implemented Evaluators should be encouraged by the school of nursmg administrators to modify the selected model to fit the nursmg education programme bemg evaluated CONCLUSION Although programme evaluahon can be time-consuming and expensive, it is necessary for development and nuuntenance of quahty nursing education programmes Escalating costs, shnnkmg appropnations, and the knowledge explosion have contnbuted to the need for examining educational programmes for efficiency and efficacy The cost of teaching nursmg students dmical skills is of particular concem smce the dmical faculty spend a great deal of time on a one-to-one basis with each student Students must also be taught more theory and psychomotor skills now than ever before, but often by the same or fewer number of faculty Administrators and faculty are accountable to consumers of nursing services, health care agenaes, students, and the institution as a whole to make sure the programme meets or exceeds acaeditation standards and that the programme is mabng the best use of scarce resources The scarcity of resouces not only involvesfisccilappropnations, but laboratory facilities as well With competition among several schools of nursing for access to the same clmical facilities, it is particularly important to utilize hospitals, nursmg homes, dimes and home health care agencies effectively so that leammg opportunities for students are maximized Bndgmg the gap between nursing education and nursmg practice is also faahtated m this way, smce programme evaluahon identifies problem areas that should be addressed There also is compehtion among schools of nursing for nursing students As a result many schools are offermg aeahve educahonal programmes m an attempt to attract students The demand and efficacy of such programmes, as well as traditional programmes, need to be conhnually evaluated An ongomg evaluahon process can help assure that mshtuhonal and programme goals are fulfilled, and establish the school's reputahon for a quality programme, which will result m attracting students Continued researdi on programme evaluahon is recommended The literature reveals a pauaty of studies
Programme evaluation Knowles L, Strozier V, Wilson J, Bodo T , Greene D & Sarver V (1985) Evaluation of a baccalaureate nursing program by alumni and of alumni by their supervisors Joumal of Nursing Education 24,261-264 Lazovik G (1979) The Report to the School of Nursing Committee on the Evaluation of Teaching (SONCET) on the Development of Student Evaluation of Teaching Scales Office for the Evaluahon of Teaching, Uruversity of Pittsburgh, Pittsburgh LincL (1987) Institutional goal analysis an approach to program evaluation Joumal of Nursmg Education 26,172-175 Litwack L, Line L & Bower D (1985) Evaluation m Nursmg Pnnaples and Practice Nahonal League for Nursing, New York Nursing schools in the future will face even greater pp 1-270 challenges for produchvity, with greater accountability for Nahonal League for Nursmg (1983) Cntena for the Evaluation of Baccalaureate and Higher Degree Programs m Nursing, 5th edn administrators and faculty A aedible, ongoing evaluation National League for Nursing, New York, pp 1-16 plan that is individualized to a particular programme is O'Neill E (1986) Comprehensive cumculum evaluation Joumal necessary in order to maintain our high standards of of Nursing Education 25, 37-39 nursmg education Patton M (1982) Practical Evaluation Sage, Beverly Hills, p 38 Posavac EJ & Carey RG (1980) Program Evaluation Methods and Case Studies Prentice-Hall, Englewood Cliffs, p 6 References PoteetG & Hodges L (1987) Strategies for ensunng a successful cumculum consultation Nurse Educator 12(5), 25—28 Anderson J (1985) Why nursing education needs curriculum Poteet G & PoUak C (1986) Increasing accountability through speaalists Nursing Outlook 3 3 , 96-97 program evaluation Nurse Educator 11(2), 4 1 - 4 7 Anderson S B , Ball S & Murphy R T (1975) Encyclopedia of Scnven M (1972) Pros and cons about goal-free evaluation Educational Evaluation Jossey-Bass, San Francisco Evaluation Comment 3,1—4 Bevis E (1982) Cumculum Building in Nurstng, 3rd edn C V Seldin P (1984) Changmg Practices in Faculty Evaluation JosseyMosby, St Louis, pp 26, 34, 35 Bass, Scin Fransisco, p 76 Clark T , Goodwin M , Manani M , Marshall M & Moore S (1983) Curriculum evaluation, an applicahon of Stufflebeam's Sohn K (1987) Program evaluation m nursmg Nurse Educator 12(2), 27-33 model in a baccalaureate school of nursing Joumal of Nursing Stakes R (1967) The countenance of educational evaluation Education 22, 54-58 Teachers College Record 68, 523-540 Fawcett J (1984) Analysis and Evaluation of Conceptual Models m Staropoli CJ & Waltz CF (1978) Developing and Evaluating Nursmg F A Davis, Philadelphia, pp 37-47 Educational Programs for Health Care Providers F A Davis, Fields M (1984) A model for comparahve course evaluation Philadelphia Joumal of Nursmg Education 2 3 , 76-78 French-Lazovik G (1974) Predictability of students' evaluahons Stewart J & Hluchyj T (1987) An approach to nursmg cumculum evaluation Nursing Outlook 35, 79-81 of college teachers from component ratings Joumal of Wakim J (1983) A workable m-depth cumculum evaluation plan Educational Psychology 66,373-385 Nurse Educator 8(3), 24-32 Horan M , Knight L, McAtee S & Weshick J (1984) A model for cumculum evaluahon Joumal of Nursing Education 2 3 , Waltz C & Bond A (1985) How can a program evaluation be comprehensive and yet cost efficient? Joumal of Nursing 319-322 Education 24, 258-261 Hulsmeyer B & Bowling A (1986) Evaluating colleagues' classWinter E, Bender A , Hertz J & Reider J (1987) Analyzing and room teaching effectiveness Nurse Educator 11(5), 19-23 evaluating a baccalaureate nursing cumculum framework Jenkins H (1986) Student participation in curriculum evaluation Nurse Educator 12(4), 10-13 Nurse Educator 11(5), 16-18
regarding the development and testing of quantitahve evaluahon tools m nursmg education Qualitahve studies are also recommended A phenomenological study exploring the meaning of a nursing educahon programme to key interest groups, for example, would provide perspectives that would be obscured in a quantitative approach that measures students' cogrutive achievement Both types of data can provide valuable insights regarding areas of strengths and weaknesses of a programme If used in complimentary fashion, both quantitahve and qualitative methods will contnbute to a comprehensive evaluation
323
