Original Paper Folia Phoniatr Logop 2014;66:89–94 DOI: 10.1159/000365354
Published online: October 17, 2014
Validity and Reliability of the European-Portuguese Preschool Language Assessment ALPE Ana Mendes a, d Marisa Lousadab, c Ana Rita Valente d Andreia Hall d, e a
School of Health Sciences, Polytechnic Institute of Setúbal (IPS), Setúbal, and b School of Health Sciences (ESSUA), c Center for Health Technology and Services Research (CINTESIS), d Institute of Electronics and Telematics Engineering of Aveiro (IEETA), and e Department of Mathematics, University of Aveiro, Aveiro, Portugal
Abstract Background/Aims: To design and standardize the Preschool Language Test (Teste de Linguagem-Avaliação da Linguagem Pré-Escolar, TL-ALPE) which assesses European Portuguese (EP)-speaking children’s receptive and expressive language abilities. Methods: To standardize TL-ALPE, data was collected on 817 EP-speaking children aged 3 years and 0 months to 5 years and 12 months. The standardization, validity and reliability procedures of TL-ALPE followed the standard recommendations. Results: TL-ALPE presented a strong cohesion (internal consistency, >80%) and strong inter- and intrajudge reliability (>90%). The content validity was verified by the description of instrument domain and the items. The construct validity of TL-ALPE was confirmed by the proportional increase in mean score values with age for both receptive and expressive abilities. The results of concurrent validity revealed strong correlations between TL-ALPE and other EP preschool language tests for expressive and total language scores. Conclusion: TL-ALPE is a valid and reliable instrument to assess language skills of EP-speaking preschool children. © 2014 S. Karger AG, Basel
© 2014 S. Karger AG, Basel 1021–7762/14/0663–0089$39.50/0 E-Mail [email protected] www.karger.com/fpl
Introduction
Children with speech and language delays represent the most common clinical cases of speech-language pathologists (SLPs). The prevalence of language delay and disorders ranges from 1 to 15% according to a systematic review of the literature [1]. Language assessment of preschool children with developmental language disorders is typically performed with standardized tests in combination with other information [2]. There are several language assessment instruments for English [3, 4]. Some of those commonly cited for preschool children are the Preschool Language Scale-4 [5], the Test of Language Development Primary (TOLD-P:4) [6], and the Clinical Evaluation of Language Fundamentals (CELF Preschool-2) [7]. For European Portuguese (EP), however, there is only one standardized instrument to assess receptive and expressive language skills of preschool-age children: the Language Assessment Test for Children (Teste de Avaliação da Linguagem na Criança, TALC) [8]. This instrument was designed to be applied in children aged 2 years and 6 months to 5 years and 11 months (2;6–5;11) and contains two subtests: (1) receptive subtest – semantics (vocabulary and semantic relations) and syntax (sentence comprehension) and (2) expressive subtest – semantics (vocabulary and semantic Marisa Lousada School of Health Sciences (ESSUA) University of Aveiro PT–3810-193 Aveiro (Portugal) E-Mail marisalousada @ ua.pt
Downloaded by: University of Exeter 149.126.76.97 - 8/14/2015 12:17:24 PM
Key Words Language · Assessment · Child language
Methods Examiners Data collection of language skills was carried out by 43 licensed and certified SLPs with experience in language testing in children. To ensure successful and consistent data collection, SLPs were provided with a detailed instruction manual containing standardization directions and scoring procedures. Pictorial materials and objects were also provided. Technical support was systematically provided on a daily basis by the research team through phone or e-mail contact. Data collection occurred in public and private kindergartens and schools.
90
Folia Phoniatr Logop 2014;66:89–94 DOI: 10.1159/000365354
Table 1. Distribution of the sample
Age group
Gender, n female
3;0 – 3;6 3;6 – 3;12 4;0 – 4;6 4;6 – 4;12 5;0 – 5;6 5;6 – 5;12 Total
Subtotal, n male
49 72 76 60 80 65
53 79 66 63 80 74
102 151 142 123 160 139
402
415
817
Sample Size and Distribution The sample size was determined through the G*Power 3 [12] program. A sample size between 850 and 900 children was indicated to detect small effect sizes in the analyses of variance (ANOVA). Thirteen districts from the Portuguese mainland and islands were selected, corresponding to four dialectal areas – northern, central-southern, Azores and Madeira [13]. Children were recruited according to the following inclusion criteria: (1) EP as native language, (2) monolingual, (3) normal language development, (4) gender, (5) age, (6) kindergarten attendance and (7) geographical location. They were selected across the Portuguese mainland as well as Madeira and the Azores based on national census distribution. Eight hundred and seventeen children were included in this study, 402 females and 415 males, with ages between 3;0 and 5;12. For each gender, age groups were tentatively equally distributed within a 6-month interval (table 1). Socioeconomic status was categorized by parental education and professional occupation. Parental education was classified according to the Portuguese levels of education: primary (1st, 2nd and 3rd cycle), secondary and university. Parental professional occupation was described through the Portuguese Classification of Occupations (Classificação Nacional de Profissões) [14]. Materials, Procedures and Scoring Ninety-two colored pictures of animals, body parts, objects, toys, food, transports and actions were designed to elicit children’s receptive and expressive language responses. Pictures were compiled and presented in a pyramidal flipped stimulus book. The picture stimulus faced the child, and the back of the pyramid had the instructions facing the examiner. Target word selection was based on EP word frequency usage described in the Portuguese Fundamental Frequency Corpus (Corpus de Frequência do Português Fundamental) [15] and the vocabulary knowledge of EP-speaking preschool children. A set of seven familiar objects (i.e., spoon, knife, fork, dish, cup, doll, car) was also used to obtain data on spatial semantic skills. TL-ALPE consisted of three sections: (1) receptive or auditory comprehension, (2) expressive or oral verbal expression, and (3) a supplemental section of metalinguistic awareness. The receptive section was used to assess and analyze the child’s auditory comprehension in semantic, morphology and syntax domains. The expressive section assessed and analyzed the child’s expressive language skills in semantic, morphology and syntax domains. The supplemental
Mendes /Lousada /Valente /Hall
Downloaded by: University of Exeter 149.126.76.97 - 8/14/2015 12:17:24 PM
awareness), morphology (word structure) and pragmatics (communicative intentions). TALC includes raw scores and percentile ranks for the receptive and expressive subtests. For the standardization, 1,002 children were recruited across Portugal. Only the internal consistency is addressed in this test, and Cronbach’s α coefficient was >0.700. There is a need for an instrument that taps a broader range of language skills (e.g., uses possessives, uses regular and irregular forms of past tense – morphological markers not assessed by TALC; builds complex sentences – use of syntax that is not analyzed by TALC; defines words, phonological awareness – metalinguistic skills not included in TALC). Furthermore, the recent literature has described specific language impairment as a heterogeneous disorder, where language domains can be selectively impaired, and different types of specific language impairment can be identified [9]; thus it is useful to have a language test that also gives standard scores for specific language domains. The Preschool Language Test (Teste de LinguagemAvaliação da Linguagem Pré-Escolar, TL-ALPE) [10] was designed to: (1) quantitatively assess the language skills of EP-speaking children aged 3;0–5;12 in receptive and expressive abilities and metalinguistic awareness in different, specific domains (a detailed description of which is presented in the Methods section); (2) qualitatively determine strengths and weaknesses in language areas; (3) document children’s progress in language following therapeutic intervention, and (4) determine intervention efficacy and efficiency. This paper aims to describe the TL-ALPE design and standardization procedures, specifically the content, construct and concurrent validity and reliability measures such as internal consistency and intra- and interjudge agreement. TL-ALPE is part of a larger research project (ALPE) that is composed of two main assessment instruments: (1) the Phonetic-Phonological Test-ALPE (Teste Fonético-Fonológico-Avaliação da Linguagem Pré-Escolar, TFF-ALPE) [11] and (2) TL-ALPE.
Table 2. Distribution of receptive and expressive tasks across language domains
Domain
Receptive
Expressive
Semantics
Identifies pictures (e.g., shoe, bed, sweat) Recognizes action in pictures (e.g., drinking, painting) Understands use of objects Understands spatial concepts (e.g., in, behind) Identifies categories (e.g., animals)
Names pictures (e.g., banana, pig, bicycle) Names actions in pictures (e.g., cooking)
Structure (morphology and syntax)
Understands basic sentences Understands complex sentences Understands passive voice sentences
Produces basic sentences Builds complex sentences Uses possessives (e.g., mine) Uses plural forms, regular and irregular Uses verb tense (e.g., regular and irregular forms of past tense, future tense, third person singular)
Semantic awareness
Defines words
Structure awareness
Recognizes morphologic and syntactic errors in sentences
Uses prepositions (e.g., on, next to) Names categories (e.g., toys) Names members of the categories (e.g., fruits)
Phonological awareness Syllable segmentation, phoneme identity
Reliability and Validity All instruments should be assessed for their validity and reliability [16–18]. The reliability of TL-ALPE was assessed, specifically internal consistency and inter- and intrajudge reliability. For internal consistency, Cronbach’s α was calculated. Interjudge reliability was assessed through the comparison of the scores obtained by two different examiners who assessed and scored the audiorecorded samples of 30 children (gender and age group equally balanced). Intrajudge reliability was assessed by comparing the scores obtained by one examiner who evaluated and scored the same audio-recorded samples of these children twice, with an interval of 5 months. After inter- and intrajudge analysis, the agreement percentages between the results were obtained. The content, construct and concurrent validities of TL-ALPE were analyzed. The content validity ensures that the instrument adequately measures the domain that it purports to measure [18].
Validity and Reliability of the Language Test ALPE
The content validity of TL-ALPE was assessed by analyzing the appropriateness of items of the skills being assessed [5]. Construct validity means that an instrument measures a theoretical construct (i.e., an explanation of a behavior based on empirical observation) [19]. The construct validity of TL-ALPE was analyzed through age differentiation (i.e., improved expressive and receptive language scores across age) [19]. Concurrent validity refers to test validity in comparison with an accepted standard instrument [18]. The concurrent validity was assessed by comparing TL-ALPE scores with TALC scores as TALC is the only EP-standardized instrument to assess receptive and expressive skills of preschool-age children. Both tests for the 30 children included in the reliability and validity analysis were administered by the same examiner. Data Analysis The sample was divided and equally balanced for age and gender. A Kolmogorov-Smirnov test with Lilliefors correction was carried out, and the results showed that the groups were not normally distributed. However, given the moderately large sample size, parametric tests were performed for comparison of means based on the central limit theorem [20]. The effects of age, gender and parents’ education and professional occupation on language scores were analyzed. One-way ANOVA and multiple comparisons were performed. The significance level (α) was 0.05. Pearson’s correlation coefficients were used to analyze concurrent validity.
Results
For the receptive and expressive sections, the number of correct answers was analyzed. Means and standard deviations, as well as maximum and minimum values were Folia Phoniatr Logop 2014;66:89–94 DOI: 10.1159/000365354
91
Downloaded by: University of Exeter 149.126.76.97 - 8/14/2015 12:17:24 PM
section collected and analyzed metalinguistic competency for semantic, morphologic, syntactic and phonological awareness. Table 2 presents the tasks assessed by TL-ALPE across language domains. Materials, verbal and model instructions, trial items and scoring directions were included in the manual. Verbal and model instructions were specific to each section. Trial items were present before administering the sections. The average testing time was 40 min for young children and 30 min for older children. All responses were recorded on the score sheet. Correct responses were indicated with a tick, and incorrect responses were transcribed. For the receptive, expressive and metalinguistic sections, the examiner scored 1 if the child’s response was correct and 0 for incorrect responses. The examiner counted a dialectal variation as correct if it was appropriate given the child’s dialectal background. The raw score was determined by the sum of the total number of correct answers rated as 1. The total score was 49 for the receptive section, 76 for the expressive section and 33 for the metalinguistic section.
total sample Age group
n
Receptive section
Expressive section
mean ± SD min max
mean ± SD min max
3;0 – 3;6 3;6 – 3;12 4;0 – 4;6 4;6 – 4;12 5;0 – 5;6 5;6 – 5;12
102 151 142 123 160 139
36.77 ± 4.65 38.73 ± 4.27 41.22 ± 3.98 41.76 ± 3.48 42.66 ± 3.39 44.02 ± 2.94
Total
817 –
25 29 29 30 32 33
46 48 48 48 49 49
39.75 ± 8.97 43.55 ± 8.81 49.44 ± 9.31 52.04 ± 8.18 55.38 ± 8.22 59.78 ± 7.62
09 20 30 26 35 34
60 66 73 74 74 73
–
–
–
–
–
calculated (table 3). The means of receptive and expressive raw scores increased as age increased. As there was a significant effect of age group (one-way ANOVA) for the receptive scores [F(5, 816) = 61.029, p < 0.001, partial η2 = 0.273, representing a large effect size (η2 > 0.1)] and for the expressive scores [F(5, 816) = 97.211, p < 0.001, partial η2 = 0.375, representing a large effect size (η2 > 0.1)], multiple comparisons were performed between the six age groups. Some of the adjacent age comparisons were not significant, but all of the comparisons between the younger and older age groups were significant for both receptive and expressive scores. As for gender, one-way ANOVA revealed that girls had significantly higher scores than boys only for the age group 5;6–5;12: F(1, 138) = 5.120, p = 0.025, partial η2 = 0.036, representing a medium effect size (0.1 > η2 > 0.01) for the receptive scores, and F(1, 138) = 4.974, p = 0.027, partial η2 = 0.035, representing a medium effect size (0.1 > η2 > 0.01) for the expressive scores. Regarding the fathers’ education, it was observed that the highest percentage was 9th grade or less (41.43%) followed by a university degree (35.31%) and high school (23.27%). For the mothers’ academic qualification, the highest number of occurrences corresponded to a university degree (48.26%) followed by 9th grade or less (28.38%) and high school (23.36%). Related to the fathers’ occupation, the majority of the sample (26.79%) was professionals, 20.99% were technicians and associate professionals and 20.65% were craft and related trades workers. Related to the mothers’ occupation, the highest percentage of the sample (40.56%) was professionals, 20.10% had elementary occupations and 14.51% were technicians and associate professionals. The effect of the fathers’ and mothers’ education and professional occupation on receptive and 92
Folia Phoniatr Logop 2014;66:89–94 DOI: 10.1159/000365354
expressive scores was analyzed. One-way ANOVAs revealed a significant effect of the mothers’ education on receptive scores [F(6, 518) = 3.367, p = 0.003, partial η2 = 0.038, representing a medium effect size (0.1 > η2 > 0.01)] and expressive scores [F(6, 518) = 3.668, p = 0.001, partial η2 = 0.041, representing a medium effect size (0.1 > η2 > 0.01)] and a significant effect of the fathers’ occupation on expressive scores [F(9, 585) = 2.200, p = 0.021, partial η2 = 0.033, representing a medium effect size (0.1 > η2 > 0.01)]. Regarding the distribution of the collected sample, the Portuguese mainland and islands were represented as follows: 62.43% central-southern, 27.28% northern, 5.88% Madeira and 4.41% Azores. Reliability and Validity The internal consistency of TL-ALPE, measured by Cronbach’s α, was 0.825 for the receptive score and 0.939 for the expressive score; Cronbach’s α for the total language score was 0.951. The interjudge agreement was 95.20% and the intrajudge agreement was 95.66%. For content validity, TL-ALPE purported to assess receptive or auditory comprehension in semantic, morphology and syntax domains; oral verbal expression in semantic, morphology and syntax domains, and metalinguistic awareness for semantic, morphologic, syntactic and phonological components of language. These language skills assessed by TL-ALPE are well documented in the scientific literature concerning language development, language disorders and psycholinguistics [1, 19, 21]. For construct validity, TL-ALPE was intended to measure preschool language ability. Scientific evidence was provided by developmental progression of raw scores with age (table 3). The data showed a significant improvement in mean values with age for both receptive and expressive scores (p < 0.05). Concurrent validity was assessed by comparing the scores of TL-ALPE and TALC [8] using Pearson’s correlation coefficients. The value obtained for the receptive section was 0.63 (p < 0.01), for the expressive section 0.85 (p < 0.01) and for the total language 0.92 (p < 0.01).
Discussion
The TL-ALPE mean values of receptive and expressive language scores increased as age increased. These results reflected the improvement across age that occurred during normal language development of 817 EP-speaking children [4]. Mendes /Lousada /Valente /Hall
Downloaded by: University of Exeter 149.126.76.97 - 8/14/2015 12:17:24 PM
Table 3. Raw scores for the receptive and expressive sections of the
This study found gender differences in language development, with the girls performing significantly better (p < 0.05) than the boys in one age group (5;6–5;11), supporting the findings of previous studies [22, 23]. The results obtained in this study also showed a significant effect of the mothers’ education on receptive and expressive development and a significant effect of the fathers’ professional occupation on expressive development (p < 0.05). These results are consistent with other studies [24, 25]. The dialectal distribution of the sample was compared with the dialectal distribution of the Portuguese population [13]: 62.43 and 60.20% (percentages of the sample and population for central-southern area); 27.28 and 34.93% (percentages of the sample and population for northern area); 5.88 and 2.54% (percentages of the sample and population for Madeira), and 4.41 and 2.34% (percentages of the sample and population for the Azores). The values obtained for the dialectal distribution of the sample are consistent with the values of the broader population. Cronbach’s α values (0.825, 0.939 and 0.951 for receptive, expressive and total language scores, respectively) indicated that TL-ALPE presented good internal consistency [26], revealing that all items of the instrument measured the same type of performance. Regarding interjudge reliability, the percentage of agreement was 95.20%, indicating strong equivalence across judges [26]. The percentage of agreement for the intrajudge reliability (95.66%) also pointed to strong stability when the same judge evaluated the test multiple times [26]. Content validity was demonstrated by the description of the test domain and the sections that compose TLALPE. The sections included in this assessment instrument (the receptive section, the expressive section and a metalinguistic awareness section) enable the SLPs to
make a detailed assessment of language skills of preschool children [26]. The construct validity of TL-ALPE was shown by the increase in means of correct answers with chronological age. The ability to use language adequately improved as age increased, with the greatest increment occurring during the preschool years. Both sections were found to have construct validity as evidenced by increases in raw scores across the six age groups (table 3) [19]. The concurrent validity was determined by the Pearson correlation coefficient between TL-ALPE and TALC. The results showed a good correlation for all comparisons (r values ranging between 0.63 and 0.92), with all values significant and indicating a large effect size (r > 0.5) [26, 27].
Conclusions
TL-ALPE is a valid and reliable instrument to assess language skills of EP-speaking children between the ages of 3;0 and 5;12. Future studies will include children with language disorders to determine how well TL-ALPE distinguishes between children with and without language disorders. Furthermore, future research is planned to develop a computerized version of TL-ALPE. Acknowledgments This work was funded by the Calouste Gulbenkian Foundation (FCG/Proc. 70964) and the Foundation for Science and Technology (FCT, FSE/CED/83549/2008). It was partially funded by FEDER through the Operational Program Competitiveness Factors – COMPETE and by National Funds through FCT in the context of the project FCOMP-01-0124-FEDER-022682 (FCT reference PEst-C/EEI/UI0127/2011) and Incentivo/EEI/UI0127/2013.
1 Law J, Boyle J, Harris F, Harkness A, Nye C: Prevalence and natural history of primary speech and language delay: findings from a recent systematic review of the literature. Int J Lang Commun Disord 2000;35:165–188. 2 Hammer CS, Pennock-Roman M, Rzasa S, Tomblin JB: An analysis of the test of language development – primary for item bias. Am J Speech Lang Pathol 2002;11:274–284. 3 Plante E, Vance R: Selection of preschool language tests. Lang Speech Hear Serv Sch 1994; 25:15–24.
Validity and Reliability of the Language Test ALPE
4 Shipley KG, McAfee JG: Assessment in Speech-Language Pathology: A Resource Manual, ed 4. New York, Delmar Cengage Learning, 2009. 5 Zimmerman IL, Steiner VG, Pond RE: Preschool Language Scale (PLS-4). San Antonio, Psychological Corporation, 2002. 6 Newcomer PL, Hammill DD: Test of Language Development: Primary (TOLD-P:4), ed 4. Austin, Pro-Ed, 2008. 7 Semel E, Wiig E, Secord W: Clinical Evaluation of Language Fundamentals Preschool-2, ed 2. London, Psychological Corporation, 2004.
8 Sua-Kay E, Tavares MD: TALC: Teste de avaliação da linguagem na criança, ed 5. Lisboa, Oficina Didáctica, 2008. 9 Friedmann N, Novogrodsky R: Subypes of SLI: SySLI, PhoSLI, LeSLI, and PraSLI; in Gavarró A, João Freitas M (eds): Language Acquisition and Development. Cambridge, Cambridge Scholars Press/CSP, 2008, pp 205–217. 10 Mendes AP, Afonso E, Lousada M, Andrade F: ALPE – avaliação de linguagem pré-escolar: um projecto em execução. XIV Semana de Fonoaudiologia/III Jornada Mauro Spinelli PUC-SP. São Paulo, 2007.
Folia Phoniatr Logop 2014;66:89–94 DOI: 10.1159/000365354
93
Downloaded by: University of Exeter 149.126.76.97 - 8/14/2015 12:17:24 PM
References
94
16 Scientific Advisory Committee of the Medical Outcome Trust: Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res 2002; 11: 193–205. 17 Lohr KN, Aaronson NK, Alonso J, Burnam MA, Patrick DL, Perrin EB, Roberts JS: Evaluation quality-of-life and health status assessment instruments: development of scientific review criteria. Clin Ther 1996;18:979–992. 18 Kerlinger FN, Lee HB: Foundations of Behavioral Research, ed 4. Fort Worth, Hartcourt College Publishers, 2000. 19 Shipley KG, McAfee JG: Assessment in Speech-Language Pathology: A Resource Manual, ed 4. Clifton Park, Delmar Cengage Learning, 2009. 20 Pring T: Research Methods in Communication Disorders. London, Whurr, 2005. 21 Bloom L, Lahey M: Language Development and Language Disorders. New York, Wiley & Sons, 1978.
Folia Phoniatr Logop 2014;66:89–94 DOI: 10.1159/000365354
22 Craig H, Washington J: Oral language expectations for African-American preschoolers and kindergartners. Am J Speech Lang Pathol 2002;11:59–70. 23 Craig H, Washington J, Thompson C: Oral language expectations for African-American children in grades 1 through 5. Am J Speech Lang Pathol 2005;14:119–130. 24 Rowe M: Child-directed speech: relation to socioeconomic status, knowledge of child development and child vocabulary skill. J Child Lang 2008;35:185–205. 25 Bates E, Marchman V, Thal D, Fenson L, Dale P, Reznick JS, Reilly J, Hartung J: Developmental and stylistic variation in the composition of early vocabulary. J Child Lang 1994;21: 85–123. 26 Almeida LS, Freire T: Metodologia da investigação em psicologia e educação, ed 4. Braga, PsiQuilibrios, 2007. 27 Field A: Discovering Statistics Using SPSS, ed 3. Los Angeles, Sage, 2010.
Mendes /Lousada /Valente /Hall
Downloaded by: University of Exeter 149.126.76.97 - 8/14/2015 12:17:24 PM
11 Lousada M, Mendes AP, Valente AR, Hall A: Standardization of a phonetic-phonological test for European Portuguese children. Folia Phoniatr Logop 2012;64:151–156. 12 Faul F, Erdfelder E, Lang AG, Buchner A: G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 2007; 39:175–191. 13 Cunha C, Cintra L: Breve gramática do português contemporâneo, ed 11. Lisboa, Edições João Sá da Costa, 1998. 14 Instituto Nacional de Estatística: Classificação nacional de profissões. INE, 1994. 15 Nascimento MF, Marques ML, Cruz ML: Português fundamental, métodos e documentos, tomo 1, inquérito de frequência. Lisboa, Instituto Nacional de Investigação Científica, Centro de Linguística da Universidade de Lisboa, 1987.
Published online: October 17, 2014
Validity and Reliability of the European-Portuguese Preschool Language Assessment ALPE Ana Mendes a, d Marisa Lousadab, c Ana Rita Valente d Andreia Hall d, e a
School of Health Sciences, Polytechnic Institute of Setúbal (IPS), Setúbal, and b School of Health Sciences (ESSUA), c Center for Health Technology and Services Research (CINTESIS), d Institute of Electronics and Telematics Engineering of Aveiro (IEETA), and e Department of Mathematics, University of Aveiro, Aveiro, Portugal
Abstract Background/Aims: To design and standardize the Preschool Language Test (Teste de Linguagem-Avaliação da Linguagem Pré-Escolar, TL-ALPE) which assesses European Portuguese (EP)-speaking children’s receptive and expressive language abilities. Methods: To standardize TL-ALPE, data was collected on 817 EP-speaking children aged 3 years and 0 months to 5 years and 12 months. The standardization, validity and reliability procedures of TL-ALPE followed the standard recommendations. Results: TL-ALPE presented a strong cohesion (internal consistency, >80%) and strong inter- and intrajudge reliability (>90%). The content validity was verified by the description of instrument domain and the items. The construct validity of TL-ALPE was confirmed by the proportional increase in mean score values with age for both receptive and expressive abilities. The results of concurrent validity revealed strong correlations between TL-ALPE and other EP preschool language tests for expressive and total language scores. Conclusion: TL-ALPE is a valid and reliable instrument to assess language skills of EP-speaking preschool children. © 2014 S. Karger AG, Basel
© 2014 S. Karger AG, Basel 1021–7762/14/0663–0089$39.50/0 E-Mail [email protected] www.karger.com/fpl
Introduction
Children with speech and language delays represent the most common clinical cases of speech-language pathologists (SLPs). The prevalence of language delay and disorders ranges from 1 to 15% according to a systematic review of the literature [1]. Language assessment of preschool children with developmental language disorders is typically performed with standardized tests in combination with other information [2]. There are several language assessment instruments for English [3, 4]. Some of those commonly cited for preschool children are the Preschool Language Scale-4 [5], the Test of Language Development Primary (TOLD-P:4) [6], and the Clinical Evaluation of Language Fundamentals (CELF Preschool-2) [7]. For European Portuguese (EP), however, there is only one standardized instrument to assess receptive and expressive language skills of preschool-age children: the Language Assessment Test for Children (Teste de Avaliação da Linguagem na Criança, TALC) [8]. This instrument was designed to be applied in children aged 2 years and 6 months to 5 years and 11 months (2;6–5;11) and contains two subtests: (1) receptive subtest – semantics (vocabulary and semantic relations) and syntax (sentence comprehension) and (2) expressive subtest – semantics (vocabulary and semantic Marisa Lousada School of Health Sciences (ESSUA) University of Aveiro PT–3810-193 Aveiro (Portugal) E-Mail marisalousada @ ua.pt
Downloaded by: University of Exeter 149.126.76.97 - 8/14/2015 12:17:24 PM
Key Words Language · Assessment · Child language
Methods Examiners Data collection of language skills was carried out by 43 licensed and certified SLPs with experience in language testing in children. To ensure successful and consistent data collection, SLPs were provided with a detailed instruction manual containing standardization directions and scoring procedures. Pictorial materials and objects were also provided. Technical support was systematically provided on a daily basis by the research team through phone or e-mail contact. Data collection occurred in public and private kindergartens and schools.
90
Folia Phoniatr Logop 2014;66:89–94 DOI: 10.1159/000365354
Table 1. Distribution of the sample
Age group
Gender, n female
3;0 – 3;6 3;6 – 3;12 4;0 – 4;6 4;6 – 4;12 5;0 – 5;6 5;6 – 5;12 Total
Subtotal, n male
49 72 76 60 80 65
53 79 66 63 80 74
102 151 142 123 160 139
402
415
817
Sample Size and Distribution The sample size was determined through the G*Power 3 [12] program. A sample size between 850 and 900 children was indicated to detect small effect sizes in the analyses of variance (ANOVA). Thirteen districts from the Portuguese mainland and islands were selected, corresponding to four dialectal areas – northern, central-southern, Azores and Madeira [13]. Children were recruited according to the following inclusion criteria: (1) EP as native language, (2) monolingual, (3) normal language development, (4) gender, (5) age, (6) kindergarten attendance and (7) geographical location. They were selected across the Portuguese mainland as well as Madeira and the Azores based on national census distribution. Eight hundred and seventeen children were included in this study, 402 females and 415 males, with ages between 3;0 and 5;12. For each gender, age groups were tentatively equally distributed within a 6-month interval (table 1). Socioeconomic status was categorized by parental education and professional occupation. Parental education was classified according to the Portuguese levels of education: primary (1st, 2nd and 3rd cycle), secondary and university. Parental professional occupation was described through the Portuguese Classification of Occupations (Classificação Nacional de Profissões) [14]. Materials, Procedures and Scoring Ninety-two colored pictures of animals, body parts, objects, toys, food, transports and actions were designed to elicit children’s receptive and expressive language responses. Pictures were compiled and presented in a pyramidal flipped stimulus book. The picture stimulus faced the child, and the back of the pyramid had the instructions facing the examiner. Target word selection was based on EP word frequency usage described in the Portuguese Fundamental Frequency Corpus (Corpus de Frequência do Português Fundamental) [15] and the vocabulary knowledge of EP-speaking preschool children. A set of seven familiar objects (i.e., spoon, knife, fork, dish, cup, doll, car) was also used to obtain data on spatial semantic skills. TL-ALPE consisted of three sections: (1) receptive or auditory comprehension, (2) expressive or oral verbal expression, and (3) a supplemental section of metalinguistic awareness. The receptive section was used to assess and analyze the child’s auditory comprehension in semantic, morphology and syntax domains. The expressive section assessed and analyzed the child’s expressive language skills in semantic, morphology and syntax domains. The supplemental
Mendes /Lousada /Valente /Hall
Downloaded by: University of Exeter 149.126.76.97 - 8/14/2015 12:17:24 PM
awareness), morphology (word structure) and pragmatics (communicative intentions). TALC includes raw scores and percentile ranks for the receptive and expressive subtests. For the standardization, 1,002 children were recruited across Portugal. Only the internal consistency is addressed in this test, and Cronbach’s α coefficient was >0.700. There is a need for an instrument that taps a broader range of language skills (e.g., uses possessives, uses regular and irregular forms of past tense – morphological markers not assessed by TALC; builds complex sentences – use of syntax that is not analyzed by TALC; defines words, phonological awareness – metalinguistic skills not included in TALC). Furthermore, the recent literature has described specific language impairment as a heterogeneous disorder, where language domains can be selectively impaired, and different types of specific language impairment can be identified [9]; thus it is useful to have a language test that also gives standard scores for specific language domains. The Preschool Language Test (Teste de LinguagemAvaliação da Linguagem Pré-Escolar, TL-ALPE) [10] was designed to: (1) quantitatively assess the language skills of EP-speaking children aged 3;0–5;12 in receptive and expressive abilities and metalinguistic awareness in different, specific domains (a detailed description of which is presented in the Methods section); (2) qualitatively determine strengths and weaknesses in language areas; (3) document children’s progress in language following therapeutic intervention, and (4) determine intervention efficacy and efficiency. This paper aims to describe the TL-ALPE design and standardization procedures, specifically the content, construct and concurrent validity and reliability measures such as internal consistency and intra- and interjudge agreement. TL-ALPE is part of a larger research project (ALPE) that is composed of two main assessment instruments: (1) the Phonetic-Phonological Test-ALPE (Teste Fonético-Fonológico-Avaliação da Linguagem Pré-Escolar, TFF-ALPE) [11] and (2) TL-ALPE.
Table 2. Distribution of receptive and expressive tasks across language domains
Domain
Receptive
Expressive
Semantics
Identifies pictures (e.g., shoe, bed, sweat) Recognizes action in pictures (e.g., drinking, painting) Understands use of objects Understands spatial concepts (e.g., in, behind) Identifies categories (e.g., animals)
Names pictures (e.g., banana, pig, bicycle) Names actions in pictures (e.g., cooking)
Structure (morphology and syntax)
Understands basic sentences Understands complex sentences Understands passive voice sentences
Produces basic sentences Builds complex sentences Uses possessives (e.g., mine) Uses plural forms, regular and irregular Uses verb tense (e.g., regular and irregular forms of past tense, future tense, third person singular)
Semantic awareness
Defines words
Structure awareness
Recognizes morphologic and syntactic errors in sentences
Uses prepositions (e.g., on, next to) Names categories (e.g., toys) Names members of the categories (e.g., fruits)
Phonological awareness Syllable segmentation, phoneme identity
Reliability and Validity All instruments should be assessed for their validity and reliability [16–18]. The reliability of TL-ALPE was assessed, specifically internal consistency and inter- and intrajudge reliability. For internal consistency, Cronbach’s α was calculated. Interjudge reliability was assessed through the comparison of the scores obtained by two different examiners who assessed and scored the audiorecorded samples of 30 children (gender and age group equally balanced). Intrajudge reliability was assessed by comparing the scores obtained by one examiner who evaluated and scored the same audio-recorded samples of these children twice, with an interval of 5 months. After inter- and intrajudge analysis, the agreement percentages between the results were obtained. The content, construct and concurrent validities of TL-ALPE were analyzed. The content validity ensures that the instrument adequately measures the domain that it purports to measure [18].
Validity and Reliability of the Language Test ALPE
The content validity of TL-ALPE was assessed by analyzing the appropriateness of items of the skills being assessed [5]. Construct validity means that an instrument measures a theoretical construct (i.e., an explanation of a behavior based on empirical observation) [19]. The construct validity of TL-ALPE was analyzed through age differentiation (i.e., improved expressive and receptive language scores across age) [19]. Concurrent validity refers to test validity in comparison with an accepted standard instrument [18]. The concurrent validity was assessed by comparing TL-ALPE scores with TALC scores as TALC is the only EP-standardized instrument to assess receptive and expressive skills of preschool-age children. Both tests for the 30 children included in the reliability and validity analysis were administered by the same examiner. Data Analysis The sample was divided and equally balanced for age and gender. A Kolmogorov-Smirnov test with Lilliefors correction was carried out, and the results showed that the groups were not normally distributed. However, given the moderately large sample size, parametric tests were performed for comparison of means based on the central limit theorem [20]. The effects of age, gender and parents’ education and professional occupation on language scores were analyzed. One-way ANOVA and multiple comparisons were performed. The significance level (α) was 0.05. Pearson’s correlation coefficients were used to analyze concurrent validity.
Results
For the receptive and expressive sections, the number of correct answers was analyzed. Means and standard deviations, as well as maximum and minimum values were Folia Phoniatr Logop 2014;66:89–94 DOI: 10.1159/000365354
91
Downloaded by: University of Exeter 149.126.76.97 - 8/14/2015 12:17:24 PM
section collected and analyzed metalinguistic competency for semantic, morphologic, syntactic and phonological awareness. Table 2 presents the tasks assessed by TL-ALPE across language domains. Materials, verbal and model instructions, trial items and scoring directions were included in the manual. Verbal and model instructions were specific to each section. Trial items were present before administering the sections. The average testing time was 40 min for young children and 30 min for older children. All responses were recorded on the score sheet. Correct responses were indicated with a tick, and incorrect responses were transcribed. For the receptive, expressive and metalinguistic sections, the examiner scored 1 if the child’s response was correct and 0 for incorrect responses. The examiner counted a dialectal variation as correct if it was appropriate given the child’s dialectal background. The raw score was determined by the sum of the total number of correct answers rated as 1. The total score was 49 for the receptive section, 76 for the expressive section and 33 for the metalinguistic section.
total sample Age group
n
Receptive section
Expressive section
mean ± SD min max
mean ± SD min max
3;0 – 3;6 3;6 – 3;12 4;0 – 4;6 4;6 – 4;12 5;0 – 5;6 5;6 – 5;12
102 151 142 123 160 139
36.77 ± 4.65 38.73 ± 4.27 41.22 ± 3.98 41.76 ± 3.48 42.66 ± 3.39 44.02 ± 2.94
Total
817 –
25 29 29 30 32 33
46 48 48 48 49 49
39.75 ± 8.97 43.55 ± 8.81 49.44 ± 9.31 52.04 ± 8.18 55.38 ± 8.22 59.78 ± 7.62
09 20 30 26 35 34
60 66 73 74 74 73
–
–
–
–
–
calculated (table 3). The means of receptive and expressive raw scores increased as age increased. As there was a significant effect of age group (one-way ANOVA) for the receptive scores [F(5, 816) = 61.029, p < 0.001, partial η2 = 0.273, representing a large effect size (η2 > 0.1)] and for the expressive scores [F(5, 816) = 97.211, p < 0.001, partial η2 = 0.375, representing a large effect size (η2 > 0.1)], multiple comparisons were performed between the six age groups. Some of the adjacent age comparisons were not significant, but all of the comparisons between the younger and older age groups were significant for both receptive and expressive scores. As for gender, one-way ANOVA revealed that girls had significantly higher scores than boys only for the age group 5;6–5;12: F(1, 138) = 5.120, p = 0.025, partial η2 = 0.036, representing a medium effect size (0.1 > η2 > 0.01) for the receptive scores, and F(1, 138) = 4.974, p = 0.027, partial η2 = 0.035, representing a medium effect size (0.1 > η2 > 0.01) for the expressive scores. Regarding the fathers’ education, it was observed that the highest percentage was 9th grade or less (41.43%) followed by a university degree (35.31%) and high school (23.27%). For the mothers’ academic qualification, the highest number of occurrences corresponded to a university degree (48.26%) followed by 9th grade or less (28.38%) and high school (23.36%). Related to the fathers’ occupation, the majority of the sample (26.79%) was professionals, 20.99% were technicians and associate professionals and 20.65% were craft and related trades workers. Related to the mothers’ occupation, the highest percentage of the sample (40.56%) was professionals, 20.10% had elementary occupations and 14.51% were technicians and associate professionals. The effect of the fathers’ and mothers’ education and professional occupation on receptive and 92
Folia Phoniatr Logop 2014;66:89–94 DOI: 10.1159/000365354
expressive scores was analyzed. One-way ANOVAs revealed a significant effect of the mothers’ education on receptive scores [F(6, 518) = 3.367, p = 0.003, partial η2 = 0.038, representing a medium effect size (0.1 > η2 > 0.01)] and expressive scores [F(6, 518) = 3.668, p = 0.001, partial η2 = 0.041, representing a medium effect size (0.1 > η2 > 0.01)] and a significant effect of the fathers’ occupation on expressive scores [F(9, 585) = 2.200, p = 0.021, partial η2 = 0.033, representing a medium effect size (0.1 > η2 > 0.01)]. Regarding the distribution of the collected sample, the Portuguese mainland and islands were represented as follows: 62.43% central-southern, 27.28% northern, 5.88% Madeira and 4.41% Azores. Reliability and Validity The internal consistency of TL-ALPE, measured by Cronbach’s α, was 0.825 for the receptive score and 0.939 for the expressive score; Cronbach’s α for the total language score was 0.951. The interjudge agreement was 95.20% and the intrajudge agreement was 95.66%. For content validity, TL-ALPE purported to assess receptive or auditory comprehension in semantic, morphology and syntax domains; oral verbal expression in semantic, morphology and syntax domains, and metalinguistic awareness for semantic, morphologic, syntactic and phonological components of language. These language skills assessed by TL-ALPE are well documented in the scientific literature concerning language development, language disorders and psycholinguistics [1, 19, 21]. For construct validity, TL-ALPE was intended to measure preschool language ability. Scientific evidence was provided by developmental progression of raw scores with age (table 3). The data showed a significant improvement in mean values with age for both receptive and expressive scores (p < 0.05). Concurrent validity was assessed by comparing the scores of TL-ALPE and TALC [8] using Pearson’s correlation coefficients. The value obtained for the receptive section was 0.63 (p < 0.01), for the expressive section 0.85 (p < 0.01) and for the total language 0.92 (p < 0.01).
Discussion
The TL-ALPE mean values of receptive and expressive language scores increased as age increased. These results reflected the improvement across age that occurred during normal language development of 817 EP-speaking children [4]. Mendes /Lousada /Valente /Hall
Downloaded by: University of Exeter 149.126.76.97 - 8/14/2015 12:17:24 PM
Table 3. Raw scores for the receptive and expressive sections of the
This study found gender differences in language development, with the girls performing significantly better (p < 0.05) than the boys in one age group (5;6–5;11), supporting the findings of previous studies [22, 23]. The results obtained in this study also showed a significant effect of the mothers’ education on receptive and expressive development and a significant effect of the fathers’ professional occupation on expressive development (p < 0.05). These results are consistent with other studies [24, 25]. The dialectal distribution of the sample was compared with the dialectal distribution of the Portuguese population [13]: 62.43 and 60.20% (percentages of the sample and population for central-southern area); 27.28 and 34.93% (percentages of the sample and population for northern area); 5.88 and 2.54% (percentages of the sample and population for Madeira), and 4.41 and 2.34% (percentages of the sample and population for the Azores). The values obtained for the dialectal distribution of the sample are consistent with the values of the broader population. Cronbach’s α values (0.825, 0.939 and 0.951 for receptive, expressive and total language scores, respectively) indicated that TL-ALPE presented good internal consistency [26], revealing that all items of the instrument measured the same type of performance. Regarding interjudge reliability, the percentage of agreement was 95.20%, indicating strong equivalence across judges [26]. The percentage of agreement for the intrajudge reliability (95.66%) also pointed to strong stability when the same judge evaluated the test multiple times [26]. Content validity was demonstrated by the description of the test domain and the sections that compose TLALPE. The sections included in this assessment instrument (the receptive section, the expressive section and a metalinguistic awareness section) enable the SLPs to
make a detailed assessment of language skills of preschool children [26]. The construct validity of TL-ALPE was shown by the increase in means of correct answers with chronological age. The ability to use language adequately improved as age increased, with the greatest increment occurring during the preschool years. Both sections were found to have construct validity as evidenced by increases in raw scores across the six age groups (table 3) [19]. The concurrent validity was determined by the Pearson correlation coefficient between TL-ALPE and TALC. The results showed a good correlation for all comparisons (r values ranging between 0.63 and 0.92), with all values significant and indicating a large effect size (r > 0.5) [26, 27].
Conclusions
TL-ALPE is a valid and reliable instrument to assess language skills of EP-speaking children between the ages of 3;0 and 5;12. Future studies will include children with language disorders to determine how well TL-ALPE distinguishes between children with and without language disorders. Furthermore, future research is planned to develop a computerized version of TL-ALPE. Acknowledgments This work was funded by the Calouste Gulbenkian Foundation (FCG/Proc. 70964) and the Foundation for Science and Technology (FCT, FSE/CED/83549/2008). It was partially funded by FEDER through the Operational Program Competitiveness Factors – COMPETE and by National Funds through FCT in the context of the project FCOMP-01-0124-FEDER-022682 (FCT reference PEst-C/EEI/UI0127/2011) and Incentivo/EEI/UI0127/2013.
1 Law J, Boyle J, Harris F, Harkness A, Nye C: Prevalence and natural history of primary speech and language delay: findings from a recent systematic review of the literature. Int J Lang Commun Disord 2000;35:165–188. 2 Hammer CS, Pennock-Roman M, Rzasa S, Tomblin JB: An analysis of the test of language development – primary for item bias. Am J Speech Lang Pathol 2002;11:274–284. 3 Plante E, Vance R: Selection of preschool language tests. Lang Speech Hear Serv Sch 1994; 25:15–24.
Validity and Reliability of the Language Test ALPE
4 Shipley KG, McAfee JG: Assessment in Speech-Language Pathology: A Resource Manual, ed 4. New York, Delmar Cengage Learning, 2009. 5 Zimmerman IL, Steiner VG, Pond RE: Preschool Language Scale (PLS-4). San Antonio, Psychological Corporation, 2002. 6 Newcomer PL, Hammill DD: Test of Language Development: Primary (TOLD-P:4), ed 4. Austin, Pro-Ed, 2008. 7 Semel E, Wiig E, Secord W: Clinical Evaluation of Language Fundamentals Preschool-2, ed 2. London, Psychological Corporation, 2004.
8 Sua-Kay E, Tavares MD: TALC: Teste de avaliação da linguagem na criança, ed 5. Lisboa, Oficina Didáctica, 2008. 9 Friedmann N, Novogrodsky R: Subypes of SLI: SySLI, PhoSLI, LeSLI, and PraSLI; in Gavarró A, João Freitas M (eds): Language Acquisition and Development. Cambridge, Cambridge Scholars Press/CSP, 2008, pp 205–217. 10 Mendes AP, Afonso E, Lousada M, Andrade F: ALPE – avaliação de linguagem pré-escolar: um projecto em execução. XIV Semana de Fonoaudiologia/III Jornada Mauro Spinelli PUC-SP. São Paulo, 2007.
Folia Phoniatr Logop 2014;66:89–94 DOI: 10.1159/000365354
93
Downloaded by: University of Exeter 149.126.76.97 - 8/14/2015 12:17:24 PM
References
94
16 Scientific Advisory Committee of the Medical Outcome Trust: Assessing health status and quality-of-life instruments: attributes and review criteria. Qual Life Res 2002; 11: 193–205. 17 Lohr KN, Aaronson NK, Alonso J, Burnam MA, Patrick DL, Perrin EB, Roberts JS: Evaluation quality-of-life and health status assessment instruments: development of scientific review criteria. Clin Ther 1996;18:979–992. 18 Kerlinger FN, Lee HB: Foundations of Behavioral Research, ed 4. Fort Worth, Hartcourt College Publishers, 2000. 19 Shipley KG, McAfee JG: Assessment in Speech-Language Pathology: A Resource Manual, ed 4. Clifton Park, Delmar Cengage Learning, 2009. 20 Pring T: Research Methods in Communication Disorders. London, Whurr, 2005. 21 Bloom L, Lahey M: Language Development and Language Disorders. New York, Wiley & Sons, 1978.
Folia Phoniatr Logop 2014;66:89–94 DOI: 10.1159/000365354
22 Craig H, Washington J: Oral language expectations for African-American preschoolers and kindergartners. Am J Speech Lang Pathol 2002;11:59–70. 23 Craig H, Washington J, Thompson C: Oral language expectations for African-American children in grades 1 through 5. Am J Speech Lang Pathol 2005;14:119–130. 24 Rowe M: Child-directed speech: relation to socioeconomic status, knowledge of child development and child vocabulary skill. J Child Lang 2008;35:185–205. 25 Bates E, Marchman V, Thal D, Fenson L, Dale P, Reznick JS, Reilly J, Hartung J: Developmental and stylistic variation in the composition of early vocabulary. J Child Lang 1994;21: 85–123. 26 Almeida LS, Freire T: Metodologia da investigação em psicologia e educação, ed 4. Braga, PsiQuilibrios, 2007. 27 Field A: Discovering Statistics Using SPSS, ed 3. Los Angeles, Sage, 2010.
Mendes /Lousada /Valente /Hall
Downloaded by: University of Exeter 149.126.76.97 - 8/14/2015 12:17:24 PM
11 Lousada M, Mendes AP, Valente AR, Hall A: Standardization of a phonetic-phonological test for European Portuguese children. Folia Phoniatr Logop 2012;64:151–156. 12 Faul F, Erdfelder E, Lang AG, Buchner A: G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 2007; 39:175–191. 13 Cunha C, Cintra L: Breve gramática do português contemporâneo, ed 11. Lisboa, Edições João Sá da Costa, 1998. 14 Instituto Nacional de Estatística: Classificação nacional de profissões. INE, 1994. 15 Nascimento MF, Marques ML, Cruz ML: Português fundamental, métodos e documentos, tomo 1, inquérito de frequência. Lisboa, Instituto Nacional de Investigação Científica, Centro de Linguística da Universidade de Lisboa, 1987.
