Nurse Education in Practice 15 (2015) 253e257
Contents lists available at ScienceDirect
Nurse Education in Practice journal homepage: www.elsevier.com/nepr
Test-retest reliability of the Clinical Learning Environment, Supervision and Nurse Teacher (CLES þ T) scale Margareta Gustafsson*, Karin Blomberg, Marie Holmefur € € School of Health and Medical Sciences, Orebro University, SE-701 82 Orebro, Sweden
a r t i c l e i n f o
a b s t r a c t
Article history: Accepted 25 February 2015
The Clinical Learning Environment, Supervision and Nurse Teacher (CLES þ T) scale evaluates the student nurses' perception of the learning environment and supervision within the clinical placement. It has never been tested in a replication study. The aim of the present study was to evaluate the test-retest reliability of the CLES þ T scale. The CLES þ T scale was administered twice to a group of 42 student nurses, with a one-week interval. Test-retest reliability was determined by calculations of Intraclass Correlation Coefficients (ICCs) and weighted Kappa coefficients. Standard Error of Measurements (SEM) and Smallest Detectable Difference (SDD) determined the precision of individual scores. BlandeAltman plots were created for analyses of systematic differences between the test occasions. The results of the study showed that the stability over time was good to excellent (ICC 0.88e0.96) in the sub-dimensions “Supervisory relationship”, “Pedagogical atmosphere on the ward” and “Role of the nurse teacher”. Measurements of “Premises of nursing on the ward” and “Leadership style of the manager” had lower but still acceptable stability (ICC 0.70e0.75). No systematic differences occurred between the test occasions. This study supports the usefulness of the CLES þ T scale as a reliable measure of the student nurses’ perception of the learning environment within the clinical placement at a hospital. © 2015 Elsevier Ltd. All rights reserved.
Keywords: Clinical learning environment Clinical education Measurement issues and assessment Measurement error Nursing education Student nurses Psychometrics Reliability Test-retest reliability
Introduction
Background
Nursing programmes include both theoretical education and clinical training. The latter takes place either in a clinical skills laboratory at the university or during clinical placements in nearby healthcare organizations. There is a need for instruments with high validity and reliability which will make it possible to find out whether clinical settings offer a good learning environment. Such instruments are important not only for the development of clinical education but also for research about nursing education. One such instrument is the Clinical Learning Environment, Supervision and Nurse Teacher (CLES þ T) scale (Saarikoski and Leino-Kilpi, 2002; Saarikoski et al., 2002, 2005, 2008). Scales should always be tested in replication studies in order to quantify the measurement error (Bland and Altman, 1999). The CLES þ T scale has gone through several psychometric tests but, as far as we know, never been tested in a replication study.
The original instrument, called the Clinical Learning Environment and Supervision (CLES) scale, was developed for the purpose to describe the student's perceptions of the clinical learning environment. The instrument was developed on the basis of an extensive literature review and tested on Finnish student nurses and thereafter on student nurses in the UK. Factor analyses supported the theoretical construct (Saarikoski and Leino-Kilpi, 2002; Saarikoski et al., 2002). The CLES scale was further tested by means of a comparison with another instrument evaluating the learning environment. The congruence between the instruments was high, supporting the validity of the CLES scale (Saarikoski et al., 2005). The CLES scale had 27 items. However, it was decided to develop an additional sub-dimension for measurement of the role of the nurse teacher. In that process, a few items in the original scale were excluded and items concerning nurse teacher's role in clinical practice were added. This new version of the instrument, the CLES þ T scale, had 34 items and had five sub-dimensions, these being “Pedagogical atmosphere on the ward”, “Leadership style of the ward manager”, “Premises of nursing on the ward”,
* Corresponding author. Tel.: þ46 19 30 36 43. E-mail address: [email protected] (M. Gustafsson). http://dx.doi.org/10.1016/j.nepr.2015.02.003 1471-5953/© 2015 Elsevier Ltd. All rights reserved.
254
M. Gustafsson et al. / Nurse Education in Practice 15 (2015) 253e257
“Supervisory relationship” and “Role of the nurse teacher” (Saarikoski et al., 2008, 2009). The CLES þ T scale has been translated and used in several European countries (Johansson et al., 2010; Papastavrou et al., 2010; Warne et al., 2010; Wolff Skaalvik et al., 2011; Henriksen et al., 2012; Bergjan and Hertel, 2013; Bos et al., 2012; Bisholt et al., 2014; Sundler et al., 2014). The Swedish version of the scale was translated by nursing teachers and back-translated by an authorized translator (Johansson et al., 2010). In the next step the scale was tested among student nurses who had completed their clinical practice mostly at a hospital. The researchers performed an exploratory factor analysis and then inter-item and itemetotal correlation analyses, and finally they tested internal consistency. Factor analysis showed that some of the items loaded on different factors among Swedish students as compared with Finnish (Saarikoski et al., 2008; Johansson et al., 2010). This applied in particular to items concerning the “Role of the nurse teacher”. Further, items in the sub-dimension “Premises of nursing on the ward” could not be separated from ones in the sub-dimension “Pedagogical atmosphere on the ward”. The conclusion was that a strong factor concerning “Premises of nursing on the ward” does not exist in the Swedish version. The factor analysis supported the finding by Saarikoski et al. (2008) that “Supervisory relationship” was the most important factor contributing to student nurses' perceptions of the clinical learning environment. The internal consistency was considered good, Cronbach's’ alpha being 0.95 for the total scale and between 0.75 and 0.96 for the sub-dimensions. The conclusion drawn from the psychometric test of the Swedish version of the CLES þ T scale was that it had satisfactory psychometric properties (Johansson et al., 2010). In order to distinguish between true change and change due to measurement error in the CLES þ T, it is necessary to determine the stability of measurements over time. This has, to our knowledge, not been done before. So the aim of the present study was to evaluate the test-retest reliability of the CLES þ T scale. Method Design and sampling The study was part of a research programme, going on 2011e2012, in which the CLES þ T scale was used for investigation of the organization of the clinical education within nursing programmes at three Swedish universities (Bisholt et al., 2014; Sundler et al., 2014). In order to determine the test-retest reliability of the scale, a questionnaire including the instrument was administered twice to a group of student nurses in spring 2011. The students were attending a nursing programme where courses including clinical practice accounted for approximately half of the time and credits. A convenience sample of student nurses in the fourth term of the nursing programme were informed about the research project, and asked whether they were willing to participate, when they had completed a course with clinical practice and were back at the university. Forty-two students gave their informed consent and filled out the questionnaires twice. Data collection Questionnaires including the CLES þ T scale were answered by the student nurses in a classroom at the university, the first one a week after they had completed their clinical practice and the second one a week later. Anonymity was maintained in that the students themselves coded the questionnaires by means of drawing a personal symbol. Questions about age, sex, location of the clinical
placement and satisfaction with the clinical placement were included in the first questionnaire. The students rated satisfaction on three items which taken together could give a score for total satisfaction with the clinical placement. The Clinical Learning Environment, Supervision and Nurse Teacher scale The CLES þ T scale consists of 34 statements divided into five sub-dimensions: “Pedagogical atmosphere on the ward” (nine items), “Leadership style of the ward manager” (four items), “Premises of nursing on the ward” (four items), “Supervisory relationship” (eight items) and “Role of the nurse teacher (NT)” (nine items). The sub-dimension “Pedagogical atmosphere on the ward” concerns the psychosocial climate and opportunities for learning, “Leadership style of the ward manager” concerns whether the ward manager is democratic and supports the staff, “Premises of nursing on the ward” concerns the quality of the nursing care and “Supervisory relationship” concerns the one-to-one relationship between the preceptor and the student (Saarikoski et al., 2002; Warne et al., 2010). Finally, the sub-dimension “Role of the nurse teacher (NT)” evaluates the teacher's pedagogical and social role in the clinical practice (Saarikoski et al., 2008). The CLES þ T items are responded to on a 5-point Likert-type scale with higher values indicating greater agreement with the statement. Response options are (1) “fully disagree”, (2) “disagree to some extent”, (3) “neither agree nor disagree”, (4) “agree to some extent” and (5) “fully agree” (Saarikoski and Leino-Kilpi, 2002; Saarikoski et al., 2002, 2005, 2008). The validity evidence of the scale based on content has been established through a review of empirical studies (n ¼ 87), literature reviews (n ¼ 5), reports of audit instruments (n ¼ 5) and discussion papers (n ¼ 7) (Saarikoski et al., 2008). Statistical analysis The students rated satisfaction with the clinical placement on three items, which taken together could give a score between 3 and 15 for total satisfaction, with a higher score indicating greater satisfaction. Mean values ranging from 1 to 5 were calculated for each sub-dimension of the CLES þ T scale. Approximately 1% of the responses to single items were missing at time 1 and time 2 and replaced with a score that did not change the student's individual mean value for the sub-dimension. Reliability is commonly expressed in a reliability coefficient ranging from 0 to 1 indicating to what extent measurements are free from measurement error. The test-retest reliability coefficient of CLES þ T was primarily determined by Intraclass Correlation Coefficients (ICCs) (Shrout and Fleiss, 1979), using a one-way analysis of variance model (i.e. ICC 1.1) at the level of individual sub-dimensions (single measures ICC for agreement). An ICC above 0.90 was considered excellent, above 0.80 was considered good and above 0.70 was considered acceptable (Vet et al., 2011). To further explore test-retest reliability, quadratic weighted Kappa coefficients were calculated for all items (Terwee, 2007). In order to analyse whether there were systematic differences between the test occasions, BlandeAltman plots were created for each subdimension of CLES þ T (Bland and Altman, 1999). Standard error of measurement (SEM) was included for determining the precision of individual scores within the subjects in the unit of test scores (Weir, 2005). The smallest detectable difference (SDD) was calculated to provide information of the smallest change that must take place between two occasions for the test to detect a real change with 95% certainty (Schreuders et al., 2003; Beckerman et al., 2001). The SDD was calculated from the SEM: SDD ¼ SEM 1.96 √2 (Schreuders et al., 2003). The statistical analyses
M. Gustafsson et al. / Nurse Education in Practice 15 (2015) 253e257
were performed using IBM SPSS Statistics 20 (IBM, Armonk, NY, USA). Ethical considerations The students received oral information in the classroom at the university before they filled out the questionnaire. They were informed about the aim of the study and assured that participation was voluntary and that an unwillingness to participate would have no consequences for their education. The codes were not revealed to the researcher. The study was part of a research programme given ethical approval by the Uppsala Regional Ethical Review Board (reg. no. 011/071) and was carried out in accordance with the ethical principles of the World Medical Association (Declaration of Helsinki). Results The student group The 42 students participating in the study had median age 23 years (interquartile range 5, min 20emax 47) and 98% were female. All of them had performed their 5-week clinical practice in a hospital. Their ratings of satisfaction with the clinical placement had mean value 12 (SD 3, range 4e15). Most of them (81%) had a score above 10, showing that they were in general very satisfied with the clinical placement. Test-retest reliability The ICC for the sub-dimensions had the range 0.70e0.96 (Table 1). The three sub-dimensions “Supervisory relationship”, “Pedagogical atmosphere on the ward” and “Role of the nurse teacher” had good to excellent test-retest reliability (ICC > 0.80). The sub-dimensions “Leadership style of the ward manager” and “Premises of nursing on the ward” displayed a lower but still acceptable stability (ICC 0.70 and 0.75 respectively). The Kappa analysis for items showed that in sub-dimensions with a lower ICC all items had similar Kappa coefficients (“Leadership style of the ward manager” kw ¼ 0.45e0.57, “Premises of nursing on the ward” kw ¼ 0.53e0.76). In sub-dimensions with higher ICC the Kappa coefficients were high (“Supervisory relationship” kw ¼ 0.77e0.89) or more fluctuating (“Pedagogical atmosphere on the ward” kw ¼ 0.61e0.85, “Role of the nurse teacher” kw ¼ 0.55e0.94). The BlandeAltman plots indicated no systematic difference between test occasions (Fig. 1). The SEM had the range 0.2e0.42, with lower precision for the two less stable sub-dimensions. The SDD, i.e. the score change required on a CLES þ T subscale to detect a change above measurement error, was 0.56e1.18 for the sub-dimensions. The SDD for the three sub-dimensions with good stability was below 1 whereas the SDD for the two poorer sub-dimensions was above 1 (Table 1).
Table 1 Intraclass correlation coefficients (ICCs), standard error of measurement (SEM) and smallest detectable difference (SDD) for test-retest of the sub-dimensions of CLES þ T.
Supervisory relationship Pedagogical atmosphere on the ward Role of the nurse teacher Leadership style of the ward manager Premises of nursing on the ward
ICC (1.1) (CI95%)
SEM
SDD
0.96 0.91 0.88 0.70 0.75
0.20 0.22 0.30 0.41 0.42
0.56 0.61 0.84 1.12 1.18
(0.93e0.98) (0.84e0.95) (0.79e0.93) (0.51e0.83) (0.58e0.86)
255
Discussion In this study, the test-retest reliability of the CLES þ T scale was evaluated with overall good results. The stability was found to be good to excellent for the sub-dimensions “Supervisory relationship”, “Pedagogical atmosphere on the ward” and “Role of the nurse teacher”. The two sub-dimensions “Leadership style of the ward manager” and “Premises of nursing on the ward” had lower but acceptable stability. The two sub-dimensions with lower reliability are in the case of the CLES þ T the shortest with four items in each sub-dimension. Generally, shorter scales are at risk of low reliability because the mean of the random errors associated with each item will be closer to zero, the more items there are in the scale (Streiner and Norman, 2008; DeVellis, 2003). Similarly, the sub-dimension “Role of the nurse teacher” may benefit from being longer (12 items) because the ICC was high despite a large range in Kappa coefficients for the items. Our interpretation, however, is that the lower reliability of the two sub-dimensions in the CLES þ T mainly has conceptual reasons. The measurements of the concepts that concern the students and their relation to the supervisor and teacher are more stable, possibly because they are related to the students' stronger personal experiences. The managers' leadership style and premises of nursing on the ward have probably less personal significance for the students during the clinical placement, which could explain why the measurements of those concepts have lower stability. The stability of the CLES þ T was evaluated by calculating ICC and Kappa coefficients, which enables comparison with other scales and gives an overall picture of stability. Reliability coefficients are not useful, however, for interpreting the scores of an individual. Therefore the scale was tested concerning precision of individual scores. Precision of individual scores, assessed by calculation of SEM, was very good for measurements of “Supervisory relationship”, “Pedagogical atmosphere on the ward” and “Role of the nurse teacher”. Also the minimum differences in repeated measurements, which could be considered as a real change not due to measurement error (SDD), were determined for each sub-dimension. If the scale is used for repeated measurements of change in the same student nurse over time, change scores larger than 0.56, 0.64 and 0.84 respectively in those sub-dimensions indicate with 95% probability that a real change has occurred. Minor change could be due to measurement error (Weir, 2005). Measurements of the sub-dimensions “Premises of nursing on the ward” and “Leadership style of the manager” were less stable over time and thus had lower precision. The results revealed that larger differences in repeated measurements are required in these subdimensions if one is to be able to reliably conclude that a real change has occurred. The sub-dimensions “Supervisory relationship”, “Pedagogical atmosphere on the ward” and “Role of the nurse teacher”, which demonstrated better stability and precision than the above-mentioned two sub-dimensions, are also reported in previous psychometric tests of the construct validity of the scale to be the strongest factors (Saarikoski et al., 2008; Johansson et al., 2010; Henriksen et al., 2012). However, taking all sub-dimensions into consideration, there was reasonable agreement between repeated measurements over time and there were no systematic differences according to the BlandeAltman plots. Test-retest studies are associated with a number of methodological challenges. A condition that has to be fulfilled is that the measurements are independent of each other (Bland and Altman, 1999). The time interval is crucial. If it is too short, there is great risk that the respondents will remember the questions and their answers from the first time. On the other hand if the time interval is too long, there is a risk that reconceptualization of the experience investigated will take place (Polit, 2014). Therefore it was decided
256
M. Gustafsson et al. / Nurse Education in Practice 15 (2015) 253e257
Fig. 1. BlandeAltman plots showing differences in scores between test occasions for each sub-dimension in CLES þ T.
M. Gustafsson et al. / Nurse Education in Practice 15 (2015) 253e257
in this study to let the student nurses respond to the questionnaire with a one-week interval. It was considered to be an interval long enough to avoid carryover effects and short enough to avoid memory distortion. Further, it is recommended that standardized administration assessments should be used in replication studies in order to avoid confounding influences (Polit, 2014). In the present study, the same researcher (MG) administered the questionnaire in a paper-and-pencil format in a classroom at the university at both time 1 and time 2. It is also recommended that heterogeneity in the sample should be ensured (Polit, 2014). The student nurses were a homogeneous group in that they all had completed the clinical practice at a hospital and were in general satisfied with the clinical placement. The majority were female, which is inevitable in studies of student nurses. However, their ages varied and they had performed their clinical placement in different hospital settings. Conclusions In summary, the present study in conjunction with previous psychometric studies supports the potential usefulness of the CLES þ T scale as a reliable measurement of the student nurses’ perception of the learning environment and the supervision during the clinical placement in a hospital. The CLES þ T scale provides educators with a tool that can indicate whether the clinical setting has shortcomings and can also be used as a measurement of change if action directed towards improvement takes place. The scale also has potential as an outcome measure in intervention studies if researchers want to test different supervision and nurse teacher models. The scale has to be further tested for assessment of the learning environment in clinical placements outside hospital. Acknowledgements We should like to express our gratitude to the student nurses who participated in this study and contributed to the development of the CLES þ T scale. References Beckerman, H., Roebroeck, M., Lankhorst, G., Becher, J., Bezemer, P., Verbeek, A., 2001. Smallest real difference, a link between reproducibility and responsiveness. Qual. Life Res. 10, 571e578. Bergjan, M., Hertel, F., 2013. Evaluating students' perception of their clinical placements d Testing the clinical learning environment and supervision and nurse teacher scale (CLESþT scale) in Germany. Nurse Educ. Today 33, 1393e1398. http://dx.doi.org/10.1016/j.nedt.2012.11.002. n, Engstrom, A., Johansson, A.S., Gustafsson, M., 2014. Bisholt, B., Ohlsson, U., Kulle Nursing students' assessment of the learning environment in different clinical settings. Nurse Educ. Pract. 304e310. http://dx.doi.org/10.1016/ j.nepr.2013.11.005. Bland, J.M., Altman, D.G., 1999. Measuring agreement in method comparison studies. Stat. Methods Med. Res. 8, 135e160. http://dx.doi.org/10.1177/ 096228029900800204.
257
Bos, E., Alinaghizadeh, H., Saarikoski, M., Kaila, P., 2012. Validating the clinical learning environment, supervision and nurse teacher CLESþT instrument in primary healthcare settings using confirmatory factor analysis. J. Clin. Nurs. 21, 1785e1788. http://dx.doi.org/10.1111/j.1365-2702.2011.04030.x. DeVellis, R.F., 2003. Scale Development Theory and Applications, second ed. Sage Publications, Inc, Thousand Oaks. Henriksen, N., Normann, H.K., Wolff Skaalvik, M., 2012. Development and testing of the norwegian version of the clinical learning environment, supervision and nurse teacher (CLESþT) evaluation scale. Int. J. Nurs. Educ. Scholarsh. 9, 1e15. http://dx.doi.org/10.1515/1548-923X.2239. Johansson, U.-B., Kaila, P., Ahlner-Elmqvist, M., Leksell, J., Isoaho, H., Saarikoski, M., 2010. Clinical learning environment, supervision and nurse teacher evaluation scale: psychometric evaluation of the Swedish version. J. Adv. Nurs. 66, 2085e2093. http://dx.doi.org/10.1111/j.1365-2648.2010.05370.x. Papastavrou, E., Lambrinou, E., Tsangari, H., Saarikoski, M., Leino-Kilpi, H., 2010. Student nurses experience of learning in the clinical environment. Nurse Educ. Pract. 10, 176e182. http://dx.doi.org/10.1016/j.nepr.2009.07.003. Polit, D.F., 2014. Getting serious about test-retest reliability: a critique of retest research and some recommendations. Qual. Life Res. http://dx.doi.org/10.1007/ s11136-014-0632-9 published online Febr. 2014. Saarikoski, M., Leino-Kilpi, 2002. The clinical learning environment and supervision by staff Nurses: developing the instrument. Int. J. Nurs. Stud. 39, 259e267. Saarikoski, M., Leino-Kilpi, H., Warne, T., 2002. Clinical learning environment and supervision: testing a research instrument in an international comparative study. Nurse Educ. Today 22, 340e349. http://dx.doi.org/10.1054/nedt.2001.0715. Saarikoski, M., Leino-Kilpi, H., Isoaho, H., Warne, T., 2005. Validation of the clinical learning environment and supervision scale. Int. J. Nurs. Educ. Scholarsh. 2, 1e16. Saarikoski, M., Isoaho, H., Warne, T., Leino-kilpi, H., 2008. The nurse teacher in clinical practice: developing the new sub-dimension to the clinical learning environment and supervision (CLES) scale. Int. J. Nurs. Stud. 45, 1233e1237. http://dx.doi.org/10.1016/j.ijnurstu.2007.07.009. Saarikoski, M., Warne, T., Kaila, P., Leino-Kilpi, H., 2009. The role of the nurse teacher in clinical practice: an empirical study of Finnish student nurse experiences. Nurse Educ. Today 29, 595e600. http://dx.doi.org/10.1016/ j.nedt.2009.01.005. Schreuders, T., Roebroeck, M., Goumans, J., van Nieuwenhuijzen, J., Stijnen, T., Stam, H., 2003. Measurement error in grip and pinch force measurements in patients with hand injuries. Phys. Ther. 83, 806e814. Shrout, P., Fleiss, J., 1979. Intraclass correlations: uses in assessing rater reliability. Psychol. Bull. 86, 420e428. Streiner, D.L., Norman, G.R., 2008. Health Measurement Scales: a Practical Guide to Their Development and Use, fourth ed. Oxford University Press, Oxford. Sundler, A.J., Bjork, M., Bisholt, B., Ohlsson, U., Engstrom, A., Gustafsson, M., 2014. Student nurses' experiences of the clinical learning environment in relation to the organization of supervision: a questionnaire survey. Nurse Educ. Today 34, 661e666. http://dx.doi.org/10.1016/j.nedt.2013.06.023. Terwee, C.B., Bot, S.D.M., de Boer, M.R., van der Windt, D.A.W.M., Knol, D.L., Dekker, J., Bouter, L.M., de Vet, H.C.W., 2007. Quality criteria were proposed for measurement properties of health status questionnaires. J. Clin. Epidemiol. 60, 34e42. Vet, H.C.W. de, Terwee, C.B., Mokkink, L.B., Knol, D.L., 2011. Measurement in Medicine: a Practical Guide. Cambridge University Press, Cambridge. Warne, T., Johansson, U.-B., Papastavrou, E., Tichelaar, E., Tomietto, M., Van den Bossche, K., Moreno, M.F.V., Saarikoski, M., 2010. An exploration of the clinical learning experience of nursing students in nine European countries. Nurse Educ. Today 30, 809e815. http://dx.doi.org/10.1016/j.nedt.2010.03.003. Wolff Skaalvik, M., Normann, H.K., Henriksen, N., 2011. Clinical learning environment and supervision: experiences of Norwegian nursing students e a questionnaire survey. J. Clin. Nurs. 20, 2294e2304. http://dx.doi.org/10.1111/j.13652702.2011.03727.x. Weir, J.P., 2005. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J. Strength Cond. Res. 19, 231e240.
Contents lists available at ScienceDirect
Nurse Education in Practice journal homepage: www.elsevier.com/nepr
Test-retest reliability of the Clinical Learning Environment, Supervision and Nurse Teacher (CLES þ T) scale Margareta Gustafsson*, Karin Blomberg, Marie Holmefur € € School of Health and Medical Sciences, Orebro University, SE-701 82 Orebro, Sweden
a r t i c l e i n f o
a b s t r a c t
Article history: Accepted 25 February 2015
The Clinical Learning Environment, Supervision and Nurse Teacher (CLES þ T) scale evaluates the student nurses' perception of the learning environment and supervision within the clinical placement. It has never been tested in a replication study. The aim of the present study was to evaluate the test-retest reliability of the CLES þ T scale. The CLES þ T scale was administered twice to a group of 42 student nurses, with a one-week interval. Test-retest reliability was determined by calculations of Intraclass Correlation Coefficients (ICCs) and weighted Kappa coefficients. Standard Error of Measurements (SEM) and Smallest Detectable Difference (SDD) determined the precision of individual scores. BlandeAltman plots were created for analyses of systematic differences between the test occasions. The results of the study showed that the stability over time was good to excellent (ICC 0.88e0.96) in the sub-dimensions “Supervisory relationship”, “Pedagogical atmosphere on the ward” and “Role of the nurse teacher”. Measurements of “Premises of nursing on the ward” and “Leadership style of the manager” had lower but still acceptable stability (ICC 0.70e0.75). No systematic differences occurred between the test occasions. This study supports the usefulness of the CLES þ T scale as a reliable measure of the student nurses’ perception of the learning environment within the clinical placement at a hospital. © 2015 Elsevier Ltd. All rights reserved.
Keywords: Clinical learning environment Clinical education Measurement issues and assessment Measurement error Nursing education Student nurses Psychometrics Reliability Test-retest reliability
Introduction
Background
Nursing programmes include both theoretical education and clinical training. The latter takes place either in a clinical skills laboratory at the university or during clinical placements in nearby healthcare organizations. There is a need for instruments with high validity and reliability which will make it possible to find out whether clinical settings offer a good learning environment. Such instruments are important not only for the development of clinical education but also for research about nursing education. One such instrument is the Clinical Learning Environment, Supervision and Nurse Teacher (CLES þ T) scale (Saarikoski and Leino-Kilpi, 2002; Saarikoski et al., 2002, 2005, 2008). Scales should always be tested in replication studies in order to quantify the measurement error (Bland and Altman, 1999). The CLES þ T scale has gone through several psychometric tests but, as far as we know, never been tested in a replication study.
The original instrument, called the Clinical Learning Environment and Supervision (CLES) scale, was developed for the purpose to describe the student's perceptions of the clinical learning environment. The instrument was developed on the basis of an extensive literature review and tested on Finnish student nurses and thereafter on student nurses in the UK. Factor analyses supported the theoretical construct (Saarikoski and Leino-Kilpi, 2002; Saarikoski et al., 2002). The CLES scale was further tested by means of a comparison with another instrument evaluating the learning environment. The congruence between the instruments was high, supporting the validity of the CLES scale (Saarikoski et al., 2005). The CLES scale had 27 items. However, it was decided to develop an additional sub-dimension for measurement of the role of the nurse teacher. In that process, a few items in the original scale were excluded and items concerning nurse teacher's role in clinical practice were added. This new version of the instrument, the CLES þ T scale, had 34 items and had five sub-dimensions, these being “Pedagogical atmosphere on the ward”, “Leadership style of the ward manager”, “Premises of nursing on the ward”,
* Corresponding author. Tel.: þ46 19 30 36 43. E-mail address: [email protected] (M. Gustafsson). http://dx.doi.org/10.1016/j.nepr.2015.02.003 1471-5953/© 2015 Elsevier Ltd. All rights reserved.
254
M. Gustafsson et al. / Nurse Education in Practice 15 (2015) 253e257
“Supervisory relationship” and “Role of the nurse teacher” (Saarikoski et al., 2008, 2009). The CLES þ T scale has been translated and used in several European countries (Johansson et al., 2010; Papastavrou et al., 2010; Warne et al., 2010; Wolff Skaalvik et al., 2011; Henriksen et al., 2012; Bergjan and Hertel, 2013; Bos et al., 2012; Bisholt et al., 2014; Sundler et al., 2014). The Swedish version of the scale was translated by nursing teachers and back-translated by an authorized translator (Johansson et al., 2010). In the next step the scale was tested among student nurses who had completed their clinical practice mostly at a hospital. The researchers performed an exploratory factor analysis and then inter-item and itemetotal correlation analyses, and finally they tested internal consistency. Factor analysis showed that some of the items loaded on different factors among Swedish students as compared with Finnish (Saarikoski et al., 2008; Johansson et al., 2010). This applied in particular to items concerning the “Role of the nurse teacher”. Further, items in the sub-dimension “Premises of nursing on the ward” could not be separated from ones in the sub-dimension “Pedagogical atmosphere on the ward”. The conclusion was that a strong factor concerning “Premises of nursing on the ward” does not exist in the Swedish version. The factor analysis supported the finding by Saarikoski et al. (2008) that “Supervisory relationship” was the most important factor contributing to student nurses' perceptions of the clinical learning environment. The internal consistency was considered good, Cronbach's’ alpha being 0.95 for the total scale and between 0.75 and 0.96 for the sub-dimensions. The conclusion drawn from the psychometric test of the Swedish version of the CLES þ T scale was that it had satisfactory psychometric properties (Johansson et al., 2010). In order to distinguish between true change and change due to measurement error in the CLES þ T, it is necessary to determine the stability of measurements over time. This has, to our knowledge, not been done before. So the aim of the present study was to evaluate the test-retest reliability of the CLES þ T scale. Method Design and sampling The study was part of a research programme, going on 2011e2012, in which the CLES þ T scale was used for investigation of the organization of the clinical education within nursing programmes at three Swedish universities (Bisholt et al., 2014; Sundler et al., 2014). In order to determine the test-retest reliability of the scale, a questionnaire including the instrument was administered twice to a group of student nurses in spring 2011. The students were attending a nursing programme where courses including clinical practice accounted for approximately half of the time and credits. A convenience sample of student nurses in the fourth term of the nursing programme were informed about the research project, and asked whether they were willing to participate, when they had completed a course with clinical practice and were back at the university. Forty-two students gave their informed consent and filled out the questionnaires twice. Data collection Questionnaires including the CLES þ T scale were answered by the student nurses in a classroom at the university, the first one a week after they had completed their clinical practice and the second one a week later. Anonymity was maintained in that the students themselves coded the questionnaires by means of drawing a personal symbol. Questions about age, sex, location of the clinical
placement and satisfaction with the clinical placement were included in the first questionnaire. The students rated satisfaction on three items which taken together could give a score for total satisfaction with the clinical placement. The Clinical Learning Environment, Supervision and Nurse Teacher scale The CLES þ T scale consists of 34 statements divided into five sub-dimensions: “Pedagogical atmosphere on the ward” (nine items), “Leadership style of the ward manager” (four items), “Premises of nursing on the ward” (four items), “Supervisory relationship” (eight items) and “Role of the nurse teacher (NT)” (nine items). The sub-dimension “Pedagogical atmosphere on the ward” concerns the psychosocial climate and opportunities for learning, “Leadership style of the ward manager” concerns whether the ward manager is democratic and supports the staff, “Premises of nursing on the ward” concerns the quality of the nursing care and “Supervisory relationship” concerns the one-to-one relationship between the preceptor and the student (Saarikoski et al., 2002; Warne et al., 2010). Finally, the sub-dimension “Role of the nurse teacher (NT)” evaluates the teacher's pedagogical and social role in the clinical practice (Saarikoski et al., 2008). The CLES þ T items are responded to on a 5-point Likert-type scale with higher values indicating greater agreement with the statement. Response options are (1) “fully disagree”, (2) “disagree to some extent”, (3) “neither agree nor disagree”, (4) “agree to some extent” and (5) “fully agree” (Saarikoski and Leino-Kilpi, 2002; Saarikoski et al., 2002, 2005, 2008). The validity evidence of the scale based on content has been established through a review of empirical studies (n ¼ 87), literature reviews (n ¼ 5), reports of audit instruments (n ¼ 5) and discussion papers (n ¼ 7) (Saarikoski et al., 2008). Statistical analysis The students rated satisfaction with the clinical placement on three items, which taken together could give a score between 3 and 15 for total satisfaction, with a higher score indicating greater satisfaction. Mean values ranging from 1 to 5 were calculated for each sub-dimension of the CLES þ T scale. Approximately 1% of the responses to single items were missing at time 1 and time 2 and replaced with a score that did not change the student's individual mean value for the sub-dimension. Reliability is commonly expressed in a reliability coefficient ranging from 0 to 1 indicating to what extent measurements are free from measurement error. The test-retest reliability coefficient of CLES þ T was primarily determined by Intraclass Correlation Coefficients (ICCs) (Shrout and Fleiss, 1979), using a one-way analysis of variance model (i.e. ICC 1.1) at the level of individual sub-dimensions (single measures ICC for agreement). An ICC above 0.90 was considered excellent, above 0.80 was considered good and above 0.70 was considered acceptable (Vet et al., 2011). To further explore test-retest reliability, quadratic weighted Kappa coefficients were calculated for all items (Terwee, 2007). In order to analyse whether there were systematic differences between the test occasions, BlandeAltman plots were created for each subdimension of CLES þ T (Bland and Altman, 1999). Standard error of measurement (SEM) was included for determining the precision of individual scores within the subjects in the unit of test scores (Weir, 2005). The smallest detectable difference (SDD) was calculated to provide information of the smallest change that must take place between two occasions for the test to detect a real change with 95% certainty (Schreuders et al., 2003; Beckerman et al., 2001). The SDD was calculated from the SEM: SDD ¼ SEM 1.96 √2 (Schreuders et al., 2003). The statistical analyses
M. Gustafsson et al. / Nurse Education in Practice 15 (2015) 253e257
were performed using IBM SPSS Statistics 20 (IBM, Armonk, NY, USA). Ethical considerations The students received oral information in the classroom at the university before they filled out the questionnaire. They were informed about the aim of the study and assured that participation was voluntary and that an unwillingness to participate would have no consequences for their education. The codes were not revealed to the researcher. The study was part of a research programme given ethical approval by the Uppsala Regional Ethical Review Board (reg. no. 011/071) and was carried out in accordance with the ethical principles of the World Medical Association (Declaration of Helsinki). Results The student group The 42 students participating in the study had median age 23 years (interquartile range 5, min 20emax 47) and 98% were female. All of them had performed their 5-week clinical practice in a hospital. Their ratings of satisfaction with the clinical placement had mean value 12 (SD 3, range 4e15). Most of them (81%) had a score above 10, showing that they were in general very satisfied with the clinical placement. Test-retest reliability The ICC for the sub-dimensions had the range 0.70e0.96 (Table 1). The three sub-dimensions “Supervisory relationship”, “Pedagogical atmosphere on the ward” and “Role of the nurse teacher” had good to excellent test-retest reliability (ICC > 0.80). The sub-dimensions “Leadership style of the ward manager” and “Premises of nursing on the ward” displayed a lower but still acceptable stability (ICC 0.70 and 0.75 respectively). The Kappa analysis for items showed that in sub-dimensions with a lower ICC all items had similar Kappa coefficients (“Leadership style of the ward manager” kw ¼ 0.45e0.57, “Premises of nursing on the ward” kw ¼ 0.53e0.76). In sub-dimensions with higher ICC the Kappa coefficients were high (“Supervisory relationship” kw ¼ 0.77e0.89) or more fluctuating (“Pedagogical atmosphere on the ward” kw ¼ 0.61e0.85, “Role of the nurse teacher” kw ¼ 0.55e0.94). The BlandeAltman plots indicated no systematic difference between test occasions (Fig. 1). The SEM had the range 0.2e0.42, with lower precision for the two less stable sub-dimensions. The SDD, i.e. the score change required on a CLES þ T subscale to detect a change above measurement error, was 0.56e1.18 for the sub-dimensions. The SDD for the three sub-dimensions with good stability was below 1 whereas the SDD for the two poorer sub-dimensions was above 1 (Table 1).
Table 1 Intraclass correlation coefficients (ICCs), standard error of measurement (SEM) and smallest detectable difference (SDD) for test-retest of the sub-dimensions of CLES þ T.
Supervisory relationship Pedagogical atmosphere on the ward Role of the nurse teacher Leadership style of the ward manager Premises of nursing on the ward
ICC (1.1) (CI95%)
SEM
SDD
0.96 0.91 0.88 0.70 0.75
0.20 0.22 0.30 0.41 0.42
0.56 0.61 0.84 1.12 1.18
(0.93e0.98) (0.84e0.95) (0.79e0.93) (0.51e0.83) (0.58e0.86)
255
Discussion In this study, the test-retest reliability of the CLES þ T scale was evaluated with overall good results. The stability was found to be good to excellent for the sub-dimensions “Supervisory relationship”, “Pedagogical atmosphere on the ward” and “Role of the nurse teacher”. The two sub-dimensions “Leadership style of the ward manager” and “Premises of nursing on the ward” had lower but acceptable stability. The two sub-dimensions with lower reliability are in the case of the CLES þ T the shortest with four items in each sub-dimension. Generally, shorter scales are at risk of low reliability because the mean of the random errors associated with each item will be closer to zero, the more items there are in the scale (Streiner and Norman, 2008; DeVellis, 2003). Similarly, the sub-dimension “Role of the nurse teacher” may benefit from being longer (12 items) because the ICC was high despite a large range in Kappa coefficients for the items. Our interpretation, however, is that the lower reliability of the two sub-dimensions in the CLES þ T mainly has conceptual reasons. The measurements of the concepts that concern the students and their relation to the supervisor and teacher are more stable, possibly because they are related to the students' stronger personal experiences. The managers' leadership style and premises of nursing on the ward have probably less personal significance for the students during the clinical placement, which could explain why the measurements of those concepts have lower stability. The stability of the CLES þ T was evaluated by calculating ICC and Kappa coefficients, which enables comparison with other scales and gives an overall picture of stability. Reliability coefficients are not useful, however, for interpreting the scores of an individual. Therefore the scale was tested concerning precision of individual scores. Precision of individual scores, assessed by calculation of SEM, was very good for measurements of “Supervisory relationship”, “Pedagogical atmosphere on the ward” and “Role of the nurse teacher”. Also the minimum differences in repeated measurements, which could be considered as a real change not due to measurement error (SDD), were determined for each sub-dimension. If the scale is used for repeated measurements of change in the same student nurse over time, change scores larger than 0.56, 0.64 and 0.84 respectively in those sub-dimensions indicate with 95% probability that a real change has occurred. Minor change could be due to measurement error (Weir, 2005). Measurements of the sub-dimensions “Premises of nursing on the ward” and “Leadership style of the manager” were less stable over time and thus had lower precision. The results revealed that larger differences in repeated measurements are required in these subdimensions if one is to be able to reliably conclude that a real change has occurred. The sub-dimensions “Supervisory relationship”, “Pedagogical atmosphere on the ward” and “Role of the nurse teacher”, which demonstrated better stability and precision than the above-mentioned two sub-dimensions, are also reported in previous psychometric tests of the construct validity of the scale to be the strongest factors (Saarikoski et al., 2008; Johansson et al., 2010; Henriksen et al., 2012). However, taking all sub-dimensions into consideration, there was reasonable agreement between repeated measurements over time and there were no systematic differences according to the BlandeAltman plots. Test-retest studies are associated with a number of methodological challenges. A condition that has to be fulfilled is that the measurements are independent of each other (Bland and Altman, 1999). The time interval is crucial. If it is too short, there is great risk that the respondents will remember the questions and their answers from the first time. On the other hand if the time interval is too long, there is a risk that reconceptualization of the experience investigated will take place (Polit, 2014). Therefore it was decided
256
M. Gustafsson et al. / Nurse Education in Practice 15 (2015) 253e257
Fig. 1. BlandeAltman plots showing differences in scores between test occasions for each sub-dimension in CLES þ T.
M. Gustafsson et al. / Nurse Education in Practice 15 (2015) 253e257
in this study to let the student nurses respond to the questionnaire with a one-week interval. It was considered to be an interval long enough to avoid carryover effects and short enough to avoid memory distortion. Further, it is recommended that standardized administration assessments should be used in replication studies in order to avoid confounding influences (Polit, 2014). In the present study, the same researcher (MG) administered the questionnaire in a paper-and-pencil format in a classroom at the university at both time 1 and time 2. It is also recommended that heterogeneity in the sample should be ensured (Polit, 2014). The student nurses were a homogeneous group in that they all had completed the clinical practice at a hospital and were in general satisfied with the clinical placement. The majority were female, which is inevitable in studies of student nurses. However, their ages varied and they had performed their clinical placement in different hospital settings. Conclusions In summary, the present study in conjunction with previous psychometric studies supports the potential usefulness of the CLES þ T scale as a reliable measurement of the student nurses’ perception of the learning environment and the supervision during the clinical placement in a hospital. The CLES þ T scale provides educators with a tool that can indicate whether the clinical setting has shortcomings and can also be used as a measurement of change if action directed towards improvement takes place. The scale also has potential as an outcome measure in intervention studies if researchers want to test different supervision and nurse teacher models. The scale has to be further tested for assessment of the learning environment in clinical placements outside hospital. Acknowledgements We should like to express our gratitude to the student nurses who participated in this study and contributed to the development of the CLES þ T scale. References Beckerman, H., Roebroeck, M., Lankhorst, G., Becher, J., Bezemer, P., Verbeek, A., 2001. Smallest real difference, a link between reproducibility and responsiveness. Qual. Life Res. 10, 571e578. Bergjan, M., Hertel, F., 2013. Evaluating students' perception of their clinical placements d Testing the clinical learning environment and supervision and nurse teacher scale (CLESþT scale) in Germany. Nurse Educ. Today 33, 1393e1398. http://dx.doi.org/10.1016/j.nedt.2012.11.002. n, Engstrom, A., Johansson, A.S., Gustafsson, M., 2014. Bisholt, B., Ohlsson, U., Kulle Nursing students' assessment of the learning environment in different clinical settings. Nurse Educ. Pract. 304e310. http://dx.doi.org/10.1016/ j.nepr.2013.11.005. Bland, J.M., Altman, D.G., 1999. Measuring agreement in method comparison studies. Stat. Methods Med. Res. 8, 135e160. http://dx.doi.org/10.1177/ 096228029900800204.
257
Bos, E., Alinaghizadeh, H., Saarikoski, M., Kaila, P., 2012. Validating the clinical learning environment, supervision and nurse teacher CLESþT instrument in primary healthcare settings using confirmatory factor analysis. J. Clin. Nurs. 21, 1785e1788. http://dx.doi.org/10.1111/j.1365-2702.2011.04030.x. DeVellis, R.F., 2003. Scale Development Theory and Applications, second ed. Sage Publications, Inc, Thousand Oaks. Henriksen, N., Normann, H.K., Wolff Skaalvik, M., 2012. Development and testing of the norwegian version of the clinical learning environment, supervision and nurse teacher (CLESþT) evaluation scale. Int. J. Nurs. Educ. Scholarsh. 9, 1e15. http://dx.doi.org/10.1515/1548-923X.2239. Johansson, U.-B., Kaila, P., Ahlner-Elmqvist, M., Leksell, J., Isoaho, H., Saarikoski, M., 2010. Clinical learning environment, supervision and nurse teacher evaluation scale: psychometric evaluation of the Swedish version. J. Adv. Nurs. 66, 2085e2093. http://dx.doi.org/10.1111/j.1365-2648.2010.05370.x. Papastavrou, E., Lambrinou, E., Tsangari, H., Saarikoski, M., Leino-Kilpi, H., 2010. Student nurses experience of learning in the clinical environment. Nurse Educ. Pract. 10, 176e182. http://dx.doi.org/10.1016/j.nepr.2009.07.003. Polit, D.F., 2014. Getting serious about test-retest reliability: a critique of retest research and some recommendations. Qual. Life Res. http://dx.doi.org/10.1007/ s11136-014-0632-9 published online Febr. 2014. Saarikoski, M., Leino-Kilpi, 2002. The clinical learning environment and supervision by staff Nurses: developing the instrument. Int. J. Nurs. Stud. 39, 259e267. Saarikoski, M., Leino-Kilpi, H., Warne, T., 2002. Clinical learning environment and supervision: testing a research instrument in an international comparative study. Nurse Educ. Today 22, 340e349. http://dx.doi.org/10.1054/nedt.2001.0715. Saarikoski, M., Leino-Kilpi, H., Isoaho, H., Warne, T., 2005. Validation of the clinical learning environment and supervision scale. Int. J. Nurs. Educ. Scholarsh. 2, 1e16. Saarikoski, M., Isoaho, H., Warne, T., Leino-kilpi, H., 2008. The nurse teacher in clinical practice: developing the new sub-dimension to the clinical learning environment and supervision (CLES) scale. Int. J. Nurs. Stud. 45, 1233e1237. http://dx.doi.org/10.1016/j.ijnurstu.2007.07.009. Saarikoski, M., Warne, T., Kaila, P., Leino-Kilpi, H., 2009. The role of the nurse teacher in clinical practice: an empirical study of Finnish student nurse experiences. Nurse Educ. Today 29, 595e600. http://dx.doi.org/10.1016/ j.nedt.2009.01.005. Schreuders, T., Roebroeck, M., Goumans, J., van Nieuwenhuijzen, J., Stijnen, T., Stam, H., 2003. Measurement error in grip and pinch force measurements in patients with hand injuries. Phys. Ther. 83, 806e814. Shrout, P., Fleiss, J., 1979. Intraclass correlations: uses in assessing rater reliability. Psychol. Bull. 86, 420e428. Streiner, D.L., Norman, G.R., 2008. Health Measurement Scales: a Practical Guide to Their Development and Use, fourth ed. Oxford University Press, Oxford. Sundler, A.J., Bjork, M., Bisholt, B., Ohlsson, U., Engstrom, A., Gustafsson, M., 2014. Student nurses' experiences of the clinical learning environment in relation to the organization of supervision: a questionnaire survey. Nurse Educ. Today 34, 661e666. http://dx.doi.org/10.1016/j.nedt.2013.06.023. Terwee, C.B., Bot, S.D.M., de Boer, M.R., van der Windt, D.A.W.M., Knol, D.L., Dekker, J., Bouter, L.M., de Vet, H.C.W., 2007. Quality criteria were proposed for measurement properties of health status questionnaires. J. Clin. Epidemiol. 60, 34e42. Vet, H.C.W. de, Terwee, C.B., Mokkink, L.B., Knol, D.L., 2011. Measurement in Medicine: a Practical Guide. Cambridge University Press, Cambridge. Warne, T., Johansson, U.-B., Papastavrou, E., Tichelaar, E., Tomietto, M., Van den Bossche, K., Moreno, M.F.V., Saarikoski, M., 2010. An exploration of the clinical learning experience of nursing students in nine European countries. Nurse Educ. Today 30, 809e815. http://dx.doi.org/10.1016/j.nedt.2010.03.003. Wolff Skaalvik, M., Normann, H.K., Henriksen, N., 2011. Clinical learning environment and supervision: experiences of Norwegian nursing students e a questionnaire survey. J. Clin. Nurs. 20, 2294e2304. http://dx.doi.org/10.1111/j.13652702.2011.03727.x. Weir, J.P., 2005. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J. Strength Cond. Res. 19, 231e240.
