European Journal of Dental Education ISSN 1396-5883

The development and validation of a questionnaire to measure the clinical learning environment for undergraduate dental students (DECLEI) A. E. Kossioni1, G. Lyrakos2, I. Ntinalexi1, R. Varela1 and I. Economu1 1 2

Dental School, University of Athens, Athens, Greece, School of Medicine, University of Athens, Athens, Greece

Keywords undergraduate dental education; dental students; clinical educational environment. Correspondence Anastassia Kossioni Department of Prosthodontics Athens Dental School Thivon 2 Goudi, 11527 Athens, Greece Tel: 00302108678464 Fax: 00302107461206 e-mail: [email protected] Accepted: 15 March 2013 doi: 10.1111/eje.12051

Abstract Aim: The aim of this study was to develop and validate according to psychometric standards a self-administered instrument to measure the students’ self-perceptions of the undergraduate clinical dental environment (DECLEI). Materials and methods: The initial questionnaire was developed using feedback from dental students, experts’ opinion and an extensive literature review. Critical incident technique (CIT) analysis was used to generate items and identify domains. Thirty clinical dental students participated in a pilot validation that generated a 67-item questionnaire. To develop a shorter and more practical version of the instrument, DECLEI67 was distributed to 153 clinical students at the University of Athens and its English version to 51 students from various dental schools, attending the 2012 European Dental Students Association meeting. This final procedure aimed to select items, identify subscales and measure internal consistency and discriminant validity. Results: A total of 202 students returned the questionnaires (response rate 99%). The final instrument included 24 items divided into three subscales: (i) organisation and learning opportunities, (ii) professionalism and communication and (iii) satisfaction and commitment to the dental studies. Cronbach’s a for the total questionnaire was 0.89. The interscale correlations ranged from 0.39 to 0.48. The instrument identified differences related to school of origin, age and duration of clinical experience. An interpretation of the scores (range 0–100) has been proposed. Conclusions: The 24-item DECLEI seemed to be a practical and valid instrument to measure a dental school’s undergraduate clinical learning environment.

Introduction Clinical training is central to undergraduate dental education and constitutes a prerequisite to be qualified to practice Dentistry in Europe. The EU Directive on the recognition of professional qualifications (2005/36/EC) demands that the undergraduate dental study programme provides the graduate dentist with the necessary skills for carrying out all activities involving prevention, diagnosis and treatment of dental patients (1). All major competences for the graduate dentist ª 2013 John Wiley & Sons A/S. Published by Blackwell Publishing Ltd Eur J Dent Educ 18 (2014) 71–79

(2) are associated with clinical education (e.g. professionalism, interpersonal, communication and social skills, clinical information gathering, diagnosis, treatment planning, therapy and prevention). According to dental students’ opinions, ‘the ideal educational environment should enable students to acquire the necessary theoretical, clinical and interpersonal competences and expose them to clinical experiences equivalent to the environment in which they are likely to be practicing dentistry after graduation’(3). An early exposure to patients, even from the first year of study, has been recommended (3, 4). 71

Dental clinical environment measurement

Efficient clinical learning requires a positive, enabling and creative interaction between the educator, the student and the educational and social environment where training takes place. Several instruments regarding the educational environment in health sciences education have been developed. The 50-item DREEM (Dundee Ready Education Environment Measure) was developed in 1997 as a universal inventory to measure health professions undergraduate educational climate (5). The 40-item PHEEM (Postgraduate Hospital Educational Environment Measure) was published by researchers in the UK in 2005 and measures the postgraduate clinical teaching and learning environment for hospital-based junior doctors (6). The Dutch Residency Educational Climate Test (D-RECT) was developed to measure the quality of clinical learning environment in graduate medical education (7). The 40-item Surgical Theatre Educational Environment Measure (STEEM) was developed in the UK to measure the quality of the learning environment in the operating theatre (8). The Clinical Learning Environment Inventory (CLEI) has been developed to assess nursing students’ perceptions of the hospital environment (9). Some previous studies have explored the dental education environment but they have used instruments originally developed for medical education, therefore they were not specific to dentistry nor were they updated (10). The Clinical Education Instruction Quality Questionnaire (ClinED IQ) originally developed for medical students (11) has been applied to US dental students (12) and recorded four major components of clinical learning: clinical learning opportunities, involvement in specific learning activities, interaction with clinical instructors and strengths and weaknesses of the programme as recorded by open-ended questions responded by the students. The Dental Student Learning Environment Survey (DSLES), also originally developed for medical education, includes 7 subscales and 55 items and has been applied in dental schools in North America (13). The DREEM instrument has been recently applied in dental schools in India (14), Greece (15), the UK (16), New Zealand (17), Pakistan (18), Germany (19), but as previously mentioned, it is a generic instrument not specific for the particular characteristics of the undergraduate dental education which has a significant clinical component. Few questionnaires have been developed to specifically study clinical dental education variables. A questionnaire to evaluate clinical dental teachers (ECDT) has been developed and validated at the University of Hong Kong (20). Gerzina et al. (21) developed a questionnaire about clinical dental teaching styles based on students’ focus groups discussion and compared their self-perceptions with those of the clinical teachers. However, there is a lack of an instrument specifically developed and validated according to current psychometric standards for measuring undergraduate dental students’ selfperceptions of the clinical educational environment. The development of such an instrument was the purpose of this study.

Materials and methods The study was approved by the Athens dental school Ethics and Research Committee (156A/2010). 72

Kossioni et al.

Step 1. Development and validation of the initial 97-item questionnaire (DECLEI-97) Content development The initial Dental Clinical Learning Environment Instrument (DECLEI) was developed using feedback from dental students, expert opinion and an extensive literature review. Critical incident technique (CIT) analysis was used to generate important items and identify domains (22). The Athens dental students were asked to identify specific factors, which positively or negatively affected their learning environment. A total of 255 dental students responded. From these interviews, elements considered as important for clinical learning environment were identified and classified into categories by the research team. Dental students from other European dental schools, members of the European Dental Students Association (EDSA), were also contacted through emails to identify items important for the clinical environment in their own schools. Further, an extensive literature review was carried out of existing publications and instruments measuring the learning environment in health sciences education to identify topics relevant to the clinical dental environment. Subscales and items relevant to the dental clinical learning environment were reviewed in the Dundee Ready Education Environment Measure-DREEM (5), the Postgraduate Hospital Educational Environment Measure-PHEEM (6), the Clinical Learning Environment Inventory for Nursing (9), the Surgical Theatre Educational Environment Measure (8), the Clinical Education Instructional Quality Questionnaire-(ClinEd IQ) (12), the questionnaire for dental clinical learning styles (21) and the report on the students’ perspectives on the academic environment agreed during the Global Congress on dental education in Dublin (3). A group of five experienced dental educators and a research psychologist, who specialises in questionnaires’ development, participated in an expert committee in the Athens dental school. This committee identified items and agreed on general factors (subscales) important to the undergraduate clinical learning environment. Ten factors were identified: (i) the clinical teachers behaviours, attitudes and teaching skills, (ii) the school’s general learning environment, (iii) the organisation of the clinics, (iv) the atmosphere in the clinics, (v) the supporting staff in the clinics, (vi) the clinical infrastructure, (vii) the communication amongst the students and between students and patients, (viii) the students’ life outside dental school, (ix) the students’ satisfaction with their dental studies and (x) the students’ personal commitment to their clinical obligations. A total of 217 items were identified and allocated to the 10 subscales. Further discussions reduced these items to 97 statements, by eliminating repetitions and those least relevant to clinical learning. Testing and validation of the initial 97-item scale (DECLEI-97) The inventory including these 97 randomly allocated statements was distributed to 30 clinical dental students in the Athens dental school (15 students with 2 years and 15 with 3 years of clinical experience). In this pilot validation procedure, students

ª 2013 John Wiley & Sons A/S. Published by Blackwell Publishing Ltd Eur J Dent Educ 18 (2014) 71–79

Kossioni et al.

were asked to score these statements on a six-item Likert scale. For ease of comparisons, all scores referred to a 100% scale: strongly disagree = 0, disagree = 20, slightly disagree = 40, slightly agree = 60, agree = 80, strongly agree = 100, with the exception of 23 negatives statements. These negative statements were included to deal with acquiescence bias (23) and were scored in reverse: strongly disagree = 100, disagree = 80, slightly disagree = 60, slightly agree = 40, agree = 20, strongly agree = 0. Ten pairs of questions had the same meaning but reverse wording to test response reliability (Pearson’s correlation coefficient ranged from 0.602 to 0.802, P  0.05). The five experts and the 30 students were asked to comment on omissions, misunderstandings, repetitions or irrelevant items to assess face validity. Descriptive response criteria were applied for each item. When  20% of the respondents did not answer an item or when a large percentage of students (  40%) provided either the lowest or highest response value, these items were considered for omission. The internal consistency coefficient (Cronbach’s a) was calculated and questions with item-total correlations of less than 0.20 were deleted. This step provided a shortened 67-item version of the questionnaire (DECLEI-67). All students found the attributes in the questionnaire relevant, but although the questionnaire was comprehensive, consistent and informative, it was considered rather long. The research team decided that the questionnaire should have fewer items to be more manageable and increase responsiveness; therefore, a new replication validation procedure was designed to omit more items.

Step 2. Validation of the 67-item scale (DECLEI67) It was decided to distribute DECLEI-67 not only to students of the Athens dental school but also to students from other nonGreek dental schools to further investigate its discriminant validity. In response, a translation of the questionnaire from Greek into English was performed. Translation procedure Τhe questionnaire was translated according to previously published criteria (24). Four bilinguals translated the questionnaire from Greek to English. A reconciliation meeting was then performed to obtain a consensus. This agreed upon English version was then back translated into Greek to detect errors of meaning and concept non-equivalence (24). The research team then met to agree on the English version of the DECLEI-67. Testing procedures The 67-item questionnaire included 52 positive and 15 negative items which had to be scored in reverse. The Greek version of the 67-item instrument was distributed to 153 clinical students of the Athens dental school who attended the 4th and the 5th year of their studies and had 2 or 3 years of clinical experience. The English version of the questionnaire was distributed to 51 clinical dental students attending the EDSA 2012 meeting in Thessaloniki, Greece. A total of ª 2013 John Wiley & Sons A/S. Published by Blackwell Publishing Ltd Eur J Dent Educ 18 (2014) 71–79

Dental clinical environment measurement

78 EDSA students had attended the meeting that year. The inclusion criteria included: having clinical experience at their home institutions and having adequate knowledge of the English language to be able to understand and respond to the questionnaire. The sample size was calculated to be representative for factor analysis, according to criteria set by Mundfrom et al. (25), with minimal sample size depending on the number of factors in the model. The psychometric properties of DECLEI were measured according to criteria previously described in detail (26). Descriptive item selection Items were considered for omission if they had high rates of non-response (  20%). Floor and ceiling effects were considered (  40%), indicated by a large percentage of students who provided either the lowest or highest response values, except for items where high values were expected. The research team decided whether those items could remain. Items related to the same concept or not directly related to dental clinical educational environment were also considered for deletion. Internal consistency Internal consistency reliability was calculated using Cronbach’s a coefficient and corrected item-total correlations were used to decide which low-contributing items should be removed from the scale. The observed scale a should be 0.70 to be acceptable, 0.80 to be good and 0.90 to be excellent (27). Factor analysis The normality of the items of the questionnaire was investigated and found to be within the level recommended for confirmatory and exploratory factor analysis (EFA) (skewness0.40, (ii) if variables were loading on the same factor they assessed the same conceptual meaning, (iii) variables loading on different components measured different conceptual meanings (30, 31). Discriminant validity Discriminant validity was measured by examining: (i) the correlations between the produced factors and between factors and 73

Dental clinical environment measurement

the total DECLEI score with Pearson’s correlation coefficient and (ii) by means of investigating the expected variance related to age, gender, duration of clinical experience and dental school of origin. Descriptive statistics and analysis between variables The descriptive statistics included means, standard deviations, and minimum and maximum values. When the assumption of normality was not seriously violated, parametric tests were used (32). As there was an increased dental school distribution of the EDSA students, they were all included in one group and compared with the Athens students. T-tests were used to investigate any variation related to: (i) gender, (ii) country of origin (Athens vs. all others) and (iii) clinical experience amongst the Athens dental students (4th vs. 5th year). Pearson’s correlation coefficient was used to investigate the correlation between DECLEI total and subscale scores and age and clinical experience. The level of statistical significance was set at P  0.05. The data were analysed using SPSS, version 19.0 (SPSS Inc., Chicago, IL, USA).

Kossioni et al.

Of the 202 students who returned the questionnaire, 119 were women and 69 men, whilst 14 did not report their gender. The mean age of the participants was 22.4  1.6 years (range: 18–32). The Athens students had been treating patients for 2.5  0.5 years (range: 2–3 years) and the non-Athens students for 2.6  1.3 years (range: 1–4).

Item selection Non-response and floor and ceiling effects led to the exclusion of 13 items (7, 11, 20, 24, 31,35, 43, 46, 48, 49, 51, 59, 60). Eleven more items were omitted by the experts (6, 33, 36, 41, 42, 45, 50, 52, 55, 61, 67) for having reverse meaning, covering the same domain or not closely relating to the clinical environment.

Internal consistency test When internal consistency reliability was performed, four items had a very low item-total correlation Q2 (r = 0.07), Q9 (r = 0.09), Q13 (r = 0.09), Q56 (r = 0.12) and were omitted from the questionnaire.

Interpretation of the scale The instrument provided the following scores: the total scale score, subscale scores and individual item scores. The item scores for each subscale were summed for each participant and the mean was calculated. The participants’ means were summed, and the mean subscale summary score was then calculated. Each of the 24 items was included in one of the three subscales. The total DECLEI score was the mean of the 24 mean item scores of all respondents. As all item scores referred to a 0–100% scale, the research team suggested an interpretation of the total DECLEI score as follows:  19.9% very poor, 20–39.9% poor, 40–59.9% moderate, 60–79.9% good,  80% excellent. The lowest score would be 0 and the highest 100. A score of  39.9% was interpreted as indicating a negative educational environment and the need for radical changes in most parameters. A moderate score (40–59.9%) was considered to suggest that the educational environment was not viewed completely negatively, but changes were necessary for improvement. The specific domains for intervention could be identified by the individual item and subscale scores. The two last scores (good and excellent,  60%) were considered to suggest that the clinical environment was positively viewed and interventions in specific areas may be needed based on the individual item analysis.

Factor analysis An initial factor analysis was then performed for the remaining items that led to seven factors explaining 51.5% of the total variance. Seven Items (3, 4, 5, 25, 40, 53, 62) had loadings lower than 0.40 and were omitted from the questionnaire. Further, six items 10, 12, 37, 39, 63, 65 loaded onto two factors and were also removed. A second factor analysis was then performed with the remaining 26 items. This final analysis identified three factors accounting for 43.8% of the variance. Two more items were removed due to loading on two factors (23, 57). The final instrument after the principal component analysis included 24 items, 20 positive and four negative (19, 28, 64, 66), that were scored in reverse (Table 1). The first factor consisted of 14 items: Q54, Q17, Q14, Q26, Q18, Q47, Q8, Q15, Q38, Q16, Q58, Q1, Q30, Q29, (eigenvalue = 5.8, variance = 22.4%), which referred to the organisation of the school enabling clinical training and the learning opportunities in the clinics. The second factor consisted of 6 items: Q32, Q19, Q21, Q27, Q64, Q22, (eigenvalue = 2.9, variance = 11.3%), which referred to the students perceptions of professionalism and communication with the patients, whilst the third consisted of four items: Q34, Q66, Q28, Q44, (eigenvalue = 2.6, variance = 10.1%) referring to satisfaction and personal commitment to dental studies.

Results Respondents

Internal consistency and discriminant validity of the final 24-item instrument

All the 153 Athens students who received the questionnaire responded to it (100% response rate). This group represented 62% of the 4th and 5th year clinical students. A total of 49 of the 51 EDSA students responded to the questionnaire (response rate: 96%). They originated from the UK, Sweden, Romania, Slovakia, Portugal, Slovenia, Turkey, France, Ireland and Serbia.

Cronbach’s a for the final 24-items questionnaire was 0.89 (Table 2). This value indicated that the total scale had excellent internal consistency. Cronbach’s a for the different subscales varied from 0.64 to 0.89. All three subscales had a moderate significant interscale correlation (0.39–0.48) and strong positive significant correlation

74

ª 2013 John Wiley & Sons A/S. Published by Blackwell Publishing Ltd Eur J Dent Educ 18 (2014) 71–79

Kossioni et al.

Dental clinical environment measurement

TABLE 1. Exploratory factor analysis (EFA) using Varimax rotation and item-total correlation for the final 24-items. The loading of the items’ on the three factors is highlighted in grey Factors Item’s number

Items

1

Q54 Q17 Q14 Q26 Q18 Q47 Q8 Q15 Q38 Q16 Q1 Q58 Q30 Q29 Q32 Q19 Q21 Q27 Q64 Q22 Q66 Q28 Q44 Q34

I have great research opportunities in my school I am learning a sufficient amount of clinical techniques In the clinics there is a feeling of mutual respect between the teachers and the students We use up-to-date materials and equipment in the clinics The clinical infrastructure of the school is satisfactory The clinical teachers are chosen with strict and proper criteria My clinical teachers are approachable The dental study programme prepared me adequately for the clinics The clinical teachers fulfil their duty and uphold the work hours of clinics I undertake patients with similar demands and difficulties as my colleagues I feel I can freely ask any question I have The clinical cases which I handle adequately prepares me for my profession The dental units’ technical problems are quickly dealt with The topics in the clinical seminars helped me in my clinical training The patients are on time for their appointments My association with my patients leads to many problems I am confident that this year I will complete my clinical responsibilities The patients are polite towards the students I am too tired to be able to work effectively in the clinics I adequately organise my patients’ folders The teachers are not adequately prepared for their class I am disappointed with my overall study experience I systematically self-evaluate my progress I am satisfied with the community service that I provide as a dentist

0.71 0.71 0.71 0.69 0.65 0.65 0.59 0.58 0.57 0.56 0.55 0.54 0.50 0.41

TABLE 2. Descriptive summary scores and Cronbach’s a for DECLEI and its subscales

Factor 1 (organisation and learning opportunities, 14 items) Factor 2 (professionalism and communication, six items) Factor 3 (satisfaction and personal commitment, four items) DECLEI total (24 items)

n

Min

Max

Mean

SD

Cronbach’s a

174

20.0

100.0

52.5

16.5

0.89

189

13.3

96.7

59.0

15.2

0.74

192

15.0

100.0

65.2

15.9

0.64

162

29.2

99.2

56.1

13.2

0.89

n = response rate in each factor. SD, standard deviation.

with the total score (Table 3). This was consistent with the results of the factor analysis that revealed three independent factors measuring different constructs. ª 2013 John Wiley & Sons A/S. Published by Blackwell Publishing Ltd Eur J Dent Educ 18 (2014) 71–79

2

3

0.33

0.38 0.38 0.38 0.64 0.62 0.59 0.57 0.50 0.48

0.33 0.70 0.64 0.57 0.45

Item-total correlation 0.56 0.68 0.66 0.57 0.59 0.52 0.59 0.63 0.53 0.48 0.58 0.57 0.34 0.38 0.31 0.40 0.55 0.24 0.37 0.36 0.32 0.38 0.29 0.35

The discriminant validity of the instrument was also explored through the observed and expected differences and similarities in relation to gender, school of origin, duration of clinical experience and age. It should be noted that there was an increased variability in the years of clinical experience for the EDSA sample as this is related to the particular curriculum of each dental school. There was a weak positive correlation between the age of the students (total sample) and: (i) DECLEI total score, (ii) subscale 1 (learning opportunities) and (iii) subscale 2 (professionalism and communication) (Table 3). When the total years of treating patients were considered, total DECLEI and subscale 1 scores showed a significant positive correlation (r = 0.23–0.27) (Table 3). Mean values, standard deviations and statistical significances for each item, subscale and total scores per gender and home institution (Athens vs. others) are presented in Table 4. The total DECLEI score was 56.1 (13.2). The good discriminant validity of the instrument was further revealed by the fact that DECLEI was able to detect significant differences between the Athens and other EDSA students (Table 4). Significant differences were recorded between the total DECLEI score, scores for subscales 1 and 2 and many individual items (Table 4). Satisfaction with their studies and commitment to their personal progress did not differ between participants of different schools. Statistical analyses did not identify any significant differences between men and women in the total score, in the three 75

Dental clinical environment measurement

Kossioni et al.

TABLE 3. Pearson product-moment correlations for DECLEI, the three factors (subscales), age and duration of clinical experience for the total sample

Factor 1 Factor 2 Factor 3 DECLEI total Age Clinical experience (years)

Factor 1

Factor 2

Factor 3

DECLEI total

Age

Clinical experience (years)

1.00 0.48** 0.40** 0.94** 0.22** 0.27**

1.00 0.39** 0.71** 0.16** 0.14

1.00 0.62** 0.05 0.12

1.00 0.21** 0.23*

1.00 0.32**

1.00

*Correlation is significant at the 0.05 level (two-tailed). **Correlation is significant at the 0.01 level (two-tailed). Significant correlations are in bold.

TABLE 4. Independent sample t-tests between sexes and University of origin (Athens dental school vs. all others) Total Mean  SD Q1 Q8 Q14 Q15 Q16 Q17 Q18 Q19 Q21 Q22 Q26 Q27 Q28 Q29 Q30 Q32 Q34 Q38 Q44 Q47 Q54 Q58 Q64 Q66 F1 F2 F3 Total

67.7 66.8 56.3 54.9 54.8 50.3 43.1 60.7 62.6 67.4 43.2 63.6 61.0 65.1 53.1 54.7 67.5 40.4 65.6 42.1 39.4 54.3 47.4 63.5 52.5 59.0 65.2 56.1

 24.3  20.8  24.2  21.9  24.5  25.2  27.6  24.7  29.1  22.4  27.2  21.3  28.8  22.1  27.2  23.8  23.3  26.5 22.8  26.5  29.8  25.3  25.4  24.1  16.5  15.3  15.9  13.2

Men Mean  SD 71.6 72.5 56.8 55.8 57.4 52.1 46.9 63.0 67.7 63.5 49.7 64.1 63.2 63.5 52.1 57.6 70.0 43.9 62.4 43.9 41.2 56.4 49.3 61.7 53.1 60.7 64.6 57.0

                           

25.1 19.7 25.4 23.2 23.4 23.8 28.2 23.9 26.6 22.6 25.3 22.5 28.0 24.2 28.1 22.3 20.9 27.0 22.7 24.7 31.9 26.0 25.3 25.8 14.6 14.6 13.2 11.2

Women Mean  SD 64.7 62.3 56.0 56.1 54.8 51.4 44.9 59.0 58.0 70.2 43.4 63.3 59.9 68.5 54.1 53.3 68.5 41.2 68.9 42.4 39.3 55.5 48.0 67.5 52.5 58.4 66.7 55.7

                           

25.1 21.3 24.5 21.9 26.2 27.3 29.2 25.5 29.4 22.3 29.0 21.3 29.8 20.5 25.9 24.9 24.4 28.0 22.2 26.5 28.6 25.6 25.9 20.9 17.6 15.6 16.6 14.3

P*

Athens Mean  SD

0.071 0.023 0.822 0.942 0.505 0.863 0.652 0.296 0.026 0.055 0.136 0.800 0.453 0.143 0.620 0.253 0.667 0.520 0.060 0.712 0.680 0.812 0.750 0.100 0.831 0.331 0.520 0.558

63.9 65.0 54.5 53.2 50.4 46.5 39.5 58.3 57.8 67.8 41.3 62.5 61.3 65.0 53.8 52.7 69.3 35.9 65.7 40.3 34.4 50.7 44.9 65.1 49.0 57.0 65.2 53.6

                           

23.7 21.1 22.7 20.5 23.8 23.8 24.9 25.3 29.4 20.8 25.7 19.8 27.4 22.4 23.9 23.2 23.3 24.8 22.4 24.3 25.1 24.8 24.2 22.6 13.7 14.9 15.2 11.4

Other Schools Mean  SD 77.1 75.0 63.8 63.8 73.8 68.3 65.1 65.1 73.5 68.6 60.4 65.1 62.2 70.4 52.2 60.9 67.7 65.3 68.4 52.0 59.6 71.1 59.2 60.4 64.8 65.7 65.3 65.7

                           

28.9 20.8 29.4 25.7 20.7 25.9 31.3 24.1 23.5 27.4 29.0 26.5 33.0 18.7 33.6 26.3 23.1 23.1 23.5 29.4 34.6 23.2 26.4 26.9 19.3 14.9 18.2 15.7

P* 0.002 0.004 0.023 0.004 0.000 0.000 0.000 0.104 0.001 0.831 0.000 0.477 0.849 0.140 0.712 0.044 0.676 0.000 0.477 0.007 0.000 0.000 0.001 0.231 0.000 0.001 0.943 0.000

*t-test. SD, standard deviation. Significant P-values are in bold.

subscales or in the individual item analyses except for two items: Q8 (My clinical teachers are approachable), and Q21 (I am confident that this year I will complete my clinical responsibilities) (Table 4). In the Athens sample which was more homogeneous in relation to their clinical experience (the students had been treating patients for either 2 or 3 years), the students’ perceptions of their clinical environment significantly improved with 76

increasing clinical experience, as indicated by the total DECLEI score and scores for subscales 1 and 2 (Table 5).

Discussion This new instrument that measured the undergraduate clinical dental environment was designed to be short, practical to use, easy to analyse, comprehensive and manageable. It can be used ª 2013 John Wiley & Sons A/S. Published by Blackwell Publishing Ltd Eur J Dent Educ 18 (2014) 71–79

Kossioni et al.

Dental clinical environment measurement

TABLE 5. Variation in DECLEI and subscales related to the duration of clinical experience in the Athens sample

Organisation and learning opportunities Professionalism and communication Satisfaction and personal commitment DECLEI total

Duration of clinical experience (years)

n

Mean

SD

P*

2 3 2 3 2 3 2 3

72 63 79 67 80 68 74 72

46.3 52.1 54.5 60.1 65.1 65.1 52.1 58.9

12.3 14.7 14.3 15.1 12.9 17.6 10.5 14.5

0.012 0.024 0.999 0.001

*t-test. n = response rate in each factor. Significant P-values are in bold.

to identify the quality of clinical education as perceived by students and the areas that need further attention. The second phase of the validation reducing the number of items was considered necessary. Shorter educational questionnaires are easier to complete, analysed faster and increase the quality and quantity of response (33). The high response rate in the current study was mainly attributed to the increased interest of the dental students in clinical practice and to the fact that the questionnaires were administered by students. Validity of an instrument refers to how well the results of a test, as interpreted for a specific purpose, can be trusted. Validity can be collected from many sources, such as content, response process, internal structure, relation to other variables and consequences (34). Various methods have been applied in the present study. The basic principles for survey design recommended by de Vaus (35) and CIT (22) were applied to generate questions and identify domains at the first stage of the development. Dental academics experienced in clinical teaching and clinical dental students were involved in the item generation procedure to ensure that the items would be relevant to the purpose of the instrument, promoting content validity (20, 34). Apart from the Athens students, the measure was also administered to students attending other European dental schools to further test content and discriminant validity. The internal structure was tested by means of reliability and factor analysis testing (34). In many previous scales, the final factors were produced by consensus clustering (5, 6, 9) and were not based on a factor analysis. The overall and observed alpha for the total scale and subscales 1 and 2 were higher than the 0.70 threshold, which is acceptable for scales (27). The relatively low Cronbach’s a value in subscale 3 can be explained by the limited number of items in this factor, which is known to affect Cronbach’s a. However, increasing the number of items would risk increasing the length of the questionnaire. The expected results of analyses between subgroups (menwomen, Athens-non-Athens students, 4–5th years Athens students) provided further evidence of acceptable discriminant validity. ª 2013 John Wiley & Sons A/S. Published by Blackwell Publishing Ltd Eur J Dent Educ 18 (2014) 71–79

DECLEI did not identify any statistically significant variation between men and women in terms of the perceptions of their clinical environment. This finding was expected as previous studies that used DREEM in both the Athens dental school (15) and in Greek medical schools (36) reported similar findings. This was also consistent with various measures from the health sciences educational environment in Europe (8, 16, 19). Gender-related variation in DREEM studies in dental schools has been occasionally recorded and may be related to the specific social environment: i.e., Pakistan (18), Saudi Arabia and Yemen (37). On the other hand, expected differences were recorded between the Athens and the other EDSA students and this further supports the discriminant validity of the instrument. As the variation in the curriculum structure and the educational philosophy varies between dental schools, such differences are expected. Increasing age and duration of clinical practice improved the students’ self-perceptions of the clinical environment. It appeared that as the students became more experienced with patients’ treatment, they felt more autonomous to practice dentistry. Senior students used the school’s learning opportunities more efficiently and were more self-confident to interact with their patients and fulfil their clinical obligations. As a result, learning became self-directed which is the basis for becoming a lifelong learner (3, 4). The findings in the literature on the effect of age and level of experience on the dental students’ perceptions of the educational environment vary. Using DREEM, some studies did not identify any significant effect of age or level of training (16), whilst opposite findings have also been reported (15, 19). However, the different purpose of DREEM compared with the present instrument precludes direct comparisons. As there is no other similar questionnaire, it has not been possible at this stage to investigate correlation with another relevant instrument (34). Subscale 1 (organisation of the school that enables clinical learning opportunities) included a large number of items including infrastructure, clinical seminars, allocation of patients cases, clinical teachers’ selection, behaviour and attitude. It was interesting that items related to teachers’ behaviour and attitudes were grouped together with items related to organisational aspects. The students seemed to view all these parameters as one factor enabling or disabling their learning opportunities. Subscale 2 (professionalism and communication with patients) identified factors related to the student’s personal involvement with the patients management and clinical responsibilities. This subscale included items related to professionalism, management skills, communication and social skills as clearly identified in the profile and competences document for the graduating European dentist (2). An interesting finding was the high rating and the stability of subscale 3, which related to the students’ satisfaction and commitment to their studies. This factor appeared to be independent from other variables that affected the clinical learning environment and may be related to the personal choice, strong motivation and commitment of dental students to study dentistry. The findings from this subscale constituted a strong basis for further improvements. 77

Dental clinical environment measurement

It is interesting that item 66 (The teachers are not adequately prepared for their class) was allocated by the factor analysis to subscale 3 (satisfaction and personal commitment). This item, although having quite a different meaning from the other items of the subscale, may indicate that the students’ satisfaction and motivation was associated with the teachers’ professional behaviour. If the teachers were not prepared for their classes, they seemed disrespectful to students and this caused disappointment. Students who are respected by their teachers also learn to respect their patients (38). Item 34 (I am satisfied with the community service that I provide as a dentist) was highly rated by all students and received the highest score by the Athens students. Learning in a humanistic environment develops important values for the healthcare professionals such as understanding and concern for others (2, 3, 38) and the dentist could become competent at displaying an appropriate caring behaviour towards patients (2). This was particularly important for the Greek students under the present circumstances of financial crisis. The students feel responsible and proud of acting as community healthcare providers. As there is no efficient public oral healthcare system in Greece, there is a large amount of patients (from the lower and middle class) who at the present time cannot afford private dental care and are treated in the dental school. An item identified as important for clinical training during item generation was the one related to the research opportunities in the dental schools (item 54, ‘I have great research opportunities in my school’). This result indicated the importance of research opportunities in the undergraduate curriculum for patient care, as scientific discovery is central to practicing evidence-based dentistry (2, 38). The response scale did not include a middle option (uncertain) to prompt students to respond to all questions and avoid the central tendency bias resulting in reduced reliability and sensitivity (15, 32, 36). Students could leave the question unanswered if they wished, but their personal involvement in the clinical training cannot justify an ‘uncertain’ option. There are some limitations in this study. At this stage, it was not possible to distribute the questionnaire to larger samples of students from other dental schools to test the international validity. Due to the small number of the EDSA students, they were all considered as one group, although there was an inherent variation between the various curricula and educational philosophies. Furthermore, English was not the native language for many of the EDSA students who responded to the English version of DECLEI, although all of the respondents claimed that their knowledge of English was satisfactory. The level of understanding of the English language of the EDSA students was not formally tested by the research team. Future studies could include: translation and validation of the instrument in different languages and dental schools to record its international validity, comparison of DECLEI with other related scales, analysis of DECLEI findings using focus groups to clarify weaknesses and related interventions, investigation of the relationship between DECLEI and patients’ satisfaction, investigation of the relationship between DECLEI and the students’ clinical assessment scores, investigation of the 78

Kossioni et al.

correlation between the scores obtained from clinical educators and those from students, and validation of the instrument in different clinical learning environments outside the dental school.

Conclusion The 24-item DECLEI has been developed and validated based on current psychometric standards. It seems to be a valid, comprehensive and practical instrument for measuring the undergraduate clinical dental environment. The scale can identify strengths and weaknesses in the dental undergraduate clinical environment and help in planning relevant curriculum changes. It can be used to compare scores between sexes, years of study, different learning environments, different institutions, and within an institution after curriculum changes have been performed. Further research is needed on the application of translated versions to different local languages in various clinical dental educational environments.

Acknowledgements The authors would like to thank the clinical educators who assisted in the development of the instrument and the students who participated in the validation procedure. They also thank the translators of the questionnaire from Greek to English.

Conflict of interest The authors report no conflict of interest.

References 1 Directive 2005/36/EC of the European Parliament and of the Council of 7 September 2005 on the recognition of professional qualifications. 2 Cowpe J, Plasschaert A, Harzer W, Vinkka-Puhakka H, Walmsley AD. Profile and competences for the graduating European dentist – update 2009. Eur J Dent Educ 2010: 14: 193–202. 3 Divaris K, Barlow PJ, Chendea SA, et al. The academic environment: the students’ perspective. Eur J Dent Educ 2008: 12 (Suppl 1): 120–130. 4 Manogue M, McLoughlin J, Christersson C, et al. Curriculum structure, content, learning and assessment in European undergraduate dental education – update 2010. Eur J Dent Educ 2011: 15: 133–141. 5 Roff S, McAleer S, Harden RM, et al. Development and validation of the Dundee Ready Education Environment Measure (DREEM). Med Teach 1997: 19: 295–299. 6 Roff S, McAleer S, Skinner A. Development and validation of an instrument to measure the postgraduate clinical learning and teaching educational environment for hospital-based junior doctors in the UK. Med Teach 2005: 27: 326–331. 7 Boor K, Van Der Vleuten C, Teunissen P, Scherpbier A, Scheele F. Development and analysis of D-RECT, an instrument measuring residents’ learning climate. Med Teach 2011: 33: 820–827. 8 Cassar K. Development of an instrument to measure the surgical operating theatre learning environment as perceived by basic surgical trainees. Med Teach 2004: 26: 260–264. 9 Chan D. Development of the Clinical Learning Environment Inventory: using the theoretical framework of learning environment

ª 2013 John Wiley & Sons A/S. Published by Blackwell Publishing Ltd Eur J Dent Educ 18 (2014) 71–79

Kossioni et al.

10

11 12

13

14

15

16

17

18

19

20

21

22 23

studies to assess nursing students’ perceptions of the hospital as a learning environment. J Nurs Educ 2002: 41: 69–75. Soemantri D, Herrera C, Riquelme A. Measuring the educational environment in health professions studies: a systematic review. Med Teach 2010: 32: 947–952. James PA, Osborne JW. A measure of medical instructional quality in ambulatory settings: the MedIQ. Fam Med 1999: 31: 263–269. Henzi D, Davis E, Jasinevicius R, Hendricson W. North American dental students’ perspectives about their clinical education. J Dent Educ 2006: 70: 361–377. Henzi D, Davis E, Jasinevicius R, Hendricson W, Cintron L, Isaacs M. Appraisal of the dental school learning environment: the students’ view. J Dent Educ 2005: 69: 1137–1147. Thomas BS, Abraham RR, Alexander M, Ramnarayan K. Students’ perceptions regarding educational environment in an Indian dental school. Med Teach 2009: 31: e185–e186. Kossioni AE, Varela R, Ekonomu I, Lyrakos G, Dimoliatis ID. Students’ perceptions of the educational environment in a Greek Dental School, as measured by DREEM. Eur J Dent Educ 2012: 16: e73–e78. Ali K, McHarg J, Kay E, et al. Academic environment in a newly established dental school with an enquiry-based curriculum: perceptions of students from the inaugural cohorts. Eur J Dent Educ 2012: 16: 102–109. Foster Page LA, Kang M, Anderson V, Thomson WM. Appraisal of the Dundee Ready Educational Environment Measure in the New Zealand dental educational environment. Eur J Dent Educ 2012: 16: 78–85. Ali K, Raja M, Watson G, Coombes L, Heffernan E. The dental school learning milieu: students’ perceptions at five academic dental institutions in Pakistan. J Dent Educ 2012: 76: 487–494. Ostapczuk MS, Hugger A, de Bruin J, Ritz-Timme S, Rotthoff T. DREEM on, dentists! Students’ perceptions of the educational environment in a German dental school as measured by the Dundee Ready Education Environment Measure. Eur J Dent Educ 2012: 16: 67–77. McGrath C, Wai Kit Yeung R, Comfort MB, McMillan AS. Development and evaluation of a questionnaire to evaluate clinical dental teachers (ECDT). Br Dent J 2005: 198: 45–48. Gerzina TM, McLean T, Fairley J. Dental clinical teaching: perceptions of students and teachers. J Dent Educ 2005: 69: 1377– 1384. Flanagan JC. The critical incident technique. Psychol Bull 1954: 51: 327–359. Fitzpatrick R. Surveys of patient satisfaction: II-Designing a questionnaire and conducting a survey. BMJ 1991: 302: 1129–1132.

ª 2013 John Wiley & Sons A/S. Published by Blackwell Publishing Ltd Eur J Dent Educ 18 (2014) 71–79

Dental clinical environment measurement

24 Medical Outcomes Trust. Trust introduces new translation criteria. Trust Bull 1997: 5: 1–4. Available from: http://www.outcomes-trust. org/bulletin/0797blltn.htm. (Accessed 3 January 2013). 25 Mundfrom DJ, Shaw DG, Tian LK. Minimum sample size recommendations for conducting factor analysis. Int J Test 2005: 5: 159–168. 26 Lyrakos GN, Vini D, Aslani H, Drosou-Servou M. Psychometric properties of the Specific Thalassemia Quality of Life Instrument for adults. Patient Prefer Adherence 2012: 6: 477–497. 27 Kline P. A Handbook of Test Construction: Introduction to Psychometric Design. New York: Methuen, 1986. 28 West SG, Finch JF, Curran PJ. Structural equation models with nonnormal variables. In: Hoyle RH, ed. Structural Equation Modeling: Concepts, Issues, and Applications. Thousand Oaks: Sage Publications, 1995: 56–75. 29 Curran PJ, West SG, Finch JF. The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychol Methods 1996: 1: 16–29. 30 Sch€ onrock-Adema J, Heijne-Penninga M, Van Hell EA, CohenSchotanus J. Necessary steps in factor analysis: enhancing validation studies of educational instruments. The PHEEM applied to clerks as an example. Med Teach 2009: 31: e226–e232. 31 Hatcher L. A Step-by-Step Approach to Using the SAS System for Factor Analysis and Structural Equation Modeling. Cary, NC: SAS Institute Inc, 1994. 32 Streiner DL, Norman GR. Health Measurement Scales – A Practical Guide to Their Development and Use. 4th edn. Oxford: Oxford University Press, 2008. 33 Nagraj S, Wall D, Jones E. The development and validation of the mini-surgical theatre educational environment measure. Med Teach 2007: 29: e192–e197. 34 Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 2006: 119: e7–e16. 35 de Vaus DA. Surveys in Social Research. London: UCL Press, 1991. 36 Dimoliatis ID, Vasilaki E, Anastassopoulos P, Ioannidis JP, Roff S. Validation of the Greek translation of the Dundee Ready Education Environment Measure (DREEM). Educ Health (Abingdon) 2010: 23: 348. 37 Al-Hazimi A, Zaini R, Al-Hyiani A, et al. Educational environment in traditional and innovative medical schools: a study in four undergraduate medical schools. Educ Health (Abingdon) 2004: 17: 192–203. 38 Haden NK, Andrieu SC, Chadwick DG, et al. The dental education environment. J Dent Educ 2006: 70: 1265–1270.

79