FINAL REPORT December 2023 PreK On My Way Pilot Study Report
Presented by: NORC at the University of Chicago
Brooke Rumper, Ph.D. Diana Serrano, Ph.D. Alicia Taylor, M.A. Zoey Merchant Jacqueline Mendez Michael Lopez, Ph.D.
Presented to: Scholastic
Amanda Alexander, Ph.D. Lindsay Marczak, M.P.A. Sarah Siegal, Ph.D. Morissa McQuide, M.S.
Final Report: Implementation and Pilot Study
2
Table of Contents
Executive Summary ...................................................................................................................... 4
Pilot Study Key Findings and Recommendations ............................................................................. 4
Introduction ....................................................................................................................................... 6
Pilot Study Summary .................................................................................................................... 7
Sample ............................................................................................................................................ 7
Method............................................................................................................................................. 9 Measures ................................................................................................................................... 9 Procedures............................................................................................................................... 32 Findings ......................................................................................................................................... 32 Descriptive Analyses ................................................................................................................ 32 Examining School Readiness Differences Between PKOMW and Comparison Students......... 33 School Readiness Measures Exhibiting Greatest Between Group Differences......................... 38 Discussion ..................................................................................................................................... 38
Conclusion ...................................................................................................................................... 40
Pilot Study Findings and Recommendations .................................................................................. 41
References ..................................................................................................................................... 43
Final Report: Implementation and Pilot Study
3
List of Tables
Table 1. Students’ Demographic Information ......................................................................................... 8
Table 2. Assessments Used in the Pilot Study and a Brief Description ................................................ 10
Table 3. Correlations Between Contextual and Demographic Characteristics and Students' Outcomes ..................................................................................................................................... 33
Table 4. Correlations Between Contextual and Demographic Characteristics and Students' Outcomes ..................................................................................................................................... 34
Table 5. Comparisons Between All PKOMW and Comparison Students.............................................. 35
Table 6. Comparisons Between All PKOMW and Comparison Students Accounting for Contextual Variables ...................................................................................................................... 35
Table 7. Descriptive Analyses of Children Identified as DLLs in PKOMW and Comparison Classrooms ..................................................................................................................................... 36
Table 8. Comparisons Between DLL PKOMW and Comparison Students Accounting for Contextual Variables ...................................................................................................................... 37
Table 9. Mean Differences Between the PKOMW and Comparison Groups ........................................ 38
Final Report: Implementation and Pilot Study
4
Executive Summary
The PreK On My Way ™ (PKOMW) evaluation consists of four phases: A case study, implementation study, pilot study and impact study. The current report presents findings from the pilot study (phase 3). The PKOMW curriculum supports PreK students in developing math, literacy, and language skills through research based, standards-aligned, interactive, and scaffolded instructional practices. It is designed to center purposeful play, socioemotional supports, hands-on activities, and culturally relevant reading materials for four and five-year-olds entering Kindergarten. The PKOMW curriculum is also designed to support multilingual learners through authentic, culturally relevant, Spanish language texts and asset-based language supports. The pilot study was a small-scale test of the methods and procedures that we plan to use in phase 4 of the evaluation, the impact study. A summary of the key study findings and recommendations from the pilot study are presented below.
Pilot Study Key Findings and Recommendations
The pilot study findings showed that:
Sample of students. Students in the PKOMW and comparison groups were similar across some demographic variables, such as age, sex, and ethnicity. But there were also some differences. The comparison group had a higher proportion of dual language learners (DLLs). Additionally, students in PKOMW classrooms came from families with significantly higher household income and had parents with higher levels of education. Both groups had the same number of children who were DLL, but the comparison classrooms had a marginally higher ( p < .10) proportion of DLL children. The term "dual language learner" (DLL) applies to young students under eight years old who are acquiring proficiency in two languages, while "multilingual language learner" refers to those learning more than two languages (Park et al., 2018). These learners form a diverse group with evolving language proficiencies influenced by exposure and support (Castro et al., 2013). All children who spoke a language other than English at home were screened using the Simon Says and Art Show subtests of the preLAS 2000 (Duncan & DeAvila, 1998). The results showed that children classified as Dual Language Learners (DLL) in the comparison group scored higher than those in the PKOMW classrooms, although the differences were not statistically significant. Classroom setting. Classrooms that implemented the PKOMW curriculum ( M = 17.10, SD = 6.56) had fewer students in their classrooms than those in comparison classrooms ( M = 19.48, SD = 1.06). Children in classrooms that implemented PKOMW outperformed those in the comparison group in expressive and receptive vocabulary and math. Analyses that included all students and did not control for demographic or other characteristics showed that students in PKOMW classrooms outperformed those in the comparison group in expressive (Expressive One-Word Vocabulary Test – 4; Frauwirth et al., 2017) (PKOMW M = 97.68, PKOMW SD = 16.09; Comparison M = 84.18, Comparison SD = 4.83) and receptive vocabulary (PPVT-5; Dunn, 2019) (PKOMW M = 90.33, PKOMW SD = 19.34;
Final Report: Implementation and Pilot Study
5
Comparison M = 73.08, Comparison SD = 6.96) and math (Woodcock-Johnson IV- Applied Problems; Shrank et al., 2018) (PKOMW M = 90.15, PKOMW SD = 20.39; Comparison M = 75.83, Comparison SD = 15.78). The unadjusted differences detected disappeared after controlling for contextual variables. After controlling for household income, parents’ level of education, number of students in the classroom, and DLL status we did not find any differences between students in PKOMW classrooms and those in the comparison classrooms on measures of expressive vocabulary (PKOMW M = 97.44, PKOMW SD = 16.52; Comparison M = 84.20, Comparison SD = 5.09), receptive vocabulary (PKOMW M = 90.30, PKOMW SD = 18.85; Comparison M = 73.73, Comparison SD = 6.91 ), math (PKOMW M = 89.74, PKOMW SD =20.86; Comparison M = 76.81, Comparison SD = 16.15 ), literacy (Woodcock-Johnson IV – Letter Word Identification; Shrank et al., 2018) (PKOMW M = 95.70, PKOMW SD =16.35; Comparison M = 93.45, Comparison SD = 14.24) , executive function (MEFS; Beck et al., 2011) (PKOMW M = 94.45, PKOMW SD = 11.51; Comparison M = 95.55, Comparison SD = 8.55), cognitive/social skills (Leiter – 3; Roid et al., 2013) (PKOMW M = 114.50, PKOMW SD = 38.11; Comparison M = 113.18, Comparison SD = 25.26) or emotions/regulation (Leiter – 3; Roid et al., 2013) skills (PKOMW M = 95.35, PKOMW SD = 36.90; Comparison M = 103.03, Comparison SD = 31.00). Significant differences were observed among DLL children in PKOMW classrooms compared to those in the comparison group. DLL students in PKOMW classrooms (n = 9) came from households with higher income, had parents with higher levels of education, and were placed in classrooms with fewer students compared to DLLs in the comparison classrooms (n = 9). DLL students in PKOMW classrooms consistently demonstrated higher scores compared to children in the comparison group across measures of expressive and receptive vocabulary, as well as math. This remained true even after adjusting for key demographic variables . Analyses focusing exclusively on DLL students and controlling for household income, parents' level of education, classroom size, and English proficiency, revealed that DLL students in PKOMW classrooms outperformed their counterparts in the comparison group in assessments of expressive (Expressive One-Word Vocabulary Test – 4; Frauwirth et al., 2017) and receptive vocabulary (Peabody Picture Vocabulary Test - 5; Dunn, 2019) and math (Woodcock-Johnson IV- Applied Problems; Shrank et al., 2018).
Based on this set of findings and insights gained from the pilot study, NORC and Scholastic should consider the following set of recommendations in preparation for the impact study:
1. The small sample size limits our ability to generalize the findings and calculate precise estimates. Strengthening recruitment efforts in collaboration with teachers. Due to the small sample size, the generalizability of our findings and the precision of our estimates are limited. To overcome this limitation, we recommend an enhanced recruitment strategy for increased student participation in the pilot study. To enhance recruitment for the upcoming impact study, especially after the pilot fell short with only 33 consent forms instead of the target 100, a collaborative approach is crucial, with a particular emphasis on involving teachers. We suggest close collaboration, clear communication of study objectives and benefits to teachers, support materials like pamphlets, and regular check-ins to address parent/guardian concerns.
Final Report: Implementation and Pilot Study
6
Involving teachers as key allies will leverage their influence, fostering effective collaboration and increasing parent participation in the upcoming impact study. 2. Assessing classroom quality and implementation fidelity. To thoroughly assess the effectiveness of the Preschool Kindergarten On My Way (PKOMW) program, we recommend conducting classroom observations in both PKOMW and comparison classrooms throughout the impact study. These observations will help evaluate the fidelity of PKOMW implementation by teachers and identify variations in teaching practices between the two groups. Understanding these differences is vital for pinpointing potential influences on school readiness measures, contributing essential data to the overall assessment of the program's impact. 3. Assessment of school readiness. Based on the pilot study's findings revealing disparities in school readiness measures, especially in the EOWPVT-4, PPVT-5, and Woodcock Johnson-IV Applied Problems subtest, we recommend the continued use of these measures in future studies due to their sensitivity to differences observed in the pilot. Additionally, for the impact study we recommend investigating specific components of the PKOMW curriculum, such as vocabulary and math elements, to enhance the interpretability of assessment outcomes. Introduction This report presents findings from the PreK On My Way ™ (PKOMW) pilot study, we examine differences in student outcomes comparing those who were in PKOMW classrooms with those who were not (referred to at the comparison group). Additionally, analyses revealed a set of recommendations, which could guide future PKOMW endeavors, including the forthcoming impact study. The PKOMW curriculum supports PreK students learning across the domains of math, literacy, and language skills through research-backed, standards-aligned, interactive, and scaffolded instruction. It emphasizes purposeful play, socioemotional support, hands-on activities, and culturally relevant reading. PKOMW includes supports dual language learners (DLLs) using authentic Spanish language text and asset-based language supports. The term dual language learners refers to young students, under the age of eight who are learning two languages, while the term multilingual language learner refers to students who are learning more than two languages (Park et al., 2018). DLLs are a diverse population where proficiencies in their languages change based on language exposure and support (Castro et al., 2013). Studies have demonstrated that supporting DLL students’ home language in the classroom is beneficial for their school readiness skills (Burchinal et al., 2012; Limlingan et al., 2020; Raikes et al., White et al., 2020). Thus, PKOMW was designed not only to meet the needs of monolingual students but those of DLLs and multilingual learners too. Authors of the PKOMW curriculum included researchers and practitioners: Tricia Zucker, PhD, from Children ’s Learning Institute at McGovern Medical School at UTHealth; Linda Mayes, MD, from Yale Child Study Center; and Jie-Qi Chen, PhD, from Erikson Institute. The overall approach of the curriculum meets state early learning standards and Head Start Early Learning Outcomes.
Final Report: Implementation and Pilot Study
7
Pilot Study Summary The PKOMW evaluation consists of four key phases. Each phase of the study has its own set of goals and seeks to answer a specific set of evaluation questions. Findings presented in this report focus on the pilot study. The pilot study's main goal was to generate critical information that will enhance the likelihood of finding a significant positive effect in a large-scale impact study, the fourth phase of this the overall evaluation. The initial pilot study was a small-scale test of the methods and procedures that we plan to use in Phase 4 of the evaluation, the impact study, and was guided by three primary research questions: • Are there differences in measures of school readiness between students in PKOMW classrooms and those in the comparison group? • Does the curriculum work better for certain groups of students (e.g., dual language learners (DLLs)?)
• Which measures demonstrated the biggest differences between groups?
To answer the research questions of interest, students instructed by PKOMW, and comparison teachers were assessed.
Sample The pilot study was conducted within a school district located in a western U.S. state. This district, referred to as School District A, is a public institution catering to a diverse student populace in a blend of urban and suburban settings. The district administers a range of schools — elementary, middle, and high — accommodating students across grade levels. With 29 schools, School District A educates approximately 25,000 students. The demographics of the students include: 2.8% White, 8.3% Black, 1% Asian or Asian/Pacific Islander, 86.1% Hispanic/Latino, 0.2% American Indian or Alaska Native, 0.4% Native Hawaiian or Pacific Islander, and 1.2% with two or more races. Furthermore, 59.1% of students qualify for federal free and reduced-price meals, and 28.9% are English language learners. All teachers in School District A are licensed, with 88.9% having three or more years of experience. The student-to-teacher ratio is 24:1, surpassing the state average. Six classrooms from schools in School District A in a state in the west coast of the U.S. participated in the PKOMW pilot study. Due to recruitment challenges and the participating district preferences, district administrators chose which teachers would implement the PKOMW curriculum, and which ones would serve as the comparison group. We were unable to determine how administrators made decisions about which teachers to assign to each group. Four classrooms implemented the PKOMW curriculum, and two classrooms participated as comparison classrooms and did not use the PKOMW curriculum. Two PKOMW preschool teachers shared that they had been teaching at their schools for 2 years or less, with 4 years of overall experience. The third PKOMW teacher had been at their current school for 4 years but had 13 years of overall experience. The fourth PKOMW teacher declined to share their level of experience. Of the two teachers in the
Final Report: Implementation and Pilot Study
8
comparison group, one had over 5 years of teaching experience, and the second had over 29 years of overall teaching experience.
The study initially aimed to recruit 100 students; however, the research team was only able to obtain parental consent for 33 students across the two conditions. See Table 1 for students’ demographic characteristics. In the overall sample students were between the ages of 42 and 63 months ( M = 56.25, SD = 5.73), 62.5% were girls, 56.3% spoke/heard Spanish at home, and 81.3% were Hispanic. There were 21 students in the PKOMW classrooms and 12 in comparison classrooms. A more detailed discussion of the key differences between the two groups is discussed below.
Table 1. Students’ Demographic Information
PKOMW Classrooms (n = 21)
Comparison Classrooms (n = 12)
Overall Sample (n = 33)
Sex
Boy
38.10%
33.30%
37.50%
Girl
61.90%
66.70%
62.50%
Home Language/DLL Status †
Spanish (DLLs)
42.90%
75.00%
56.30%
English
52.40%
25.00%
40.60%
Ethnicity
Hispanic or Latino
81.00%
83.30%
81.30%
Not Hispanic or Latino
14.30%
16.70%
15.60%
Race
Asian
0
9.50%
6.30%
American Indian or Alaskan Native
0
4.80%
3.10%
Black
0
4.80%
3.10%
White
47.60%
50.00%
50.00%
Two or more races
66.70%
16.70%
6.30%
No Response
33.30%
33.00%
28.10%
Household Income*
Below $15,000
0
9.50%
9.40%
$15,000-$25, 000
4.80%
8.30%
9.40%
$25, 001-$35, 000
4.80%
16.70%
9.30%
$35, 001-$50,000
4.80%
16.70%
15.60%
$50,001-$65,000
14.30%
33.30%
9.40%
Final Report: Implementation and Pilot Study
9
PKOMW Classrooms (n = 21)
Comparison Classrooms (n = 12)
Overall Sample (n = 33)
$65,001 or more
57.10%
16.70%
43.80%
Parent Highest Education**
8th grade or less
0
8.30%
3.10%
9th grade- 12th grade no diploma
4.80%
16.70%
9.40%
High school graduate or GED
19.00%
41.70%
25.00%
Some college but no degree
19.00%
16.70%
18.80%
Associates degree (AA or AS)
4.80%
8.30%
6.30%
Bachelor’s degree (BA, BS, or AB)
14.30%
8.30%
12.50%
Graduate or professional degree
0
33.30%
21.90%
M
SD
M
SD
M
SD
Age
57.62
4.02
54.83
7.64
56.25
5.73
Average number of students enrolled in class
17.10
6.56
23.67**
0.49
19.48
1.06
Note. Demographic information is missing for one child. The overall sample contained 18 DLL children. † marginally higher, p <.10
*significant difference, p < .05 **significant difference p < .01 ***significant difference, p < .001
Method
Measures
For the pilot study, the NORC team used multiple measures to assess differences between students in the PKOMW classrooms and those in the comparison group. We were interested in measures of language proficiency, expressive and receptive vocabulary, literacy, math, executive functioning, and social emotional skills. Strong early skills in these areas have been associated with long-term academic success (Bleses et al., 2016; Duncan et al., 2007; Quirk et al., 2017). Below we describe the measures that we used (Table 2). In addition to identifying differences, if any, between the two groups we also sought to test our procedures and the use of the assessments in preparation for the impact study.
Final Report: Pilot Study
10
Table 2 . Assessments Used in the Pilot Study and a Brief Description
Domain
Measure
Description
Reliability
Duration
Language screener
preLAS 2000 (Duncan & De Avila, 1998)
DLL students’ English language skills are screened using the Art Show and Simon Says subtests. Art Show screens students’ expressive language skills and Simons Says screens their receptive skills. Both subtests contain 10 items. During the Art Show students are shown a series of pictures and they are asked to name the item and in later trials asked to tell what the item does. In the Simon Says subtest, students are asked to do complete a series of tasks (e.g., “Simon says turn the paper over”) The PPVT-5 assesses students’ English receptive vocabulary skills. Students are shown pictures of several items and asked to point to a particular item. The EOWPVT-4 is a measure of students’ English expressive vocabulary and the EOWPVT-SBE is a measure of students’ Spanish expressive vocabulary. During this assessment students are presented with pictures and asked to name items and actions. Students’ literacy skills were measured using the Letter-Word Identification or Identificación de letras y palabras subtests of the Woodcock-Johnson IV and Batería IV, respectively. During this assessment students are presented with a testbook and asked to identify letters and to read words.
~10 minutes
Art Show: = .90; Simon Says: = .88
Receptive Vocabulary
Peabody Picture Vocabulary Test – 5 (PPVT-5; Dunn, 2019)
For students between 3-6-years-old = .97- .98
~ 10-15 minutes
Expressive Vocabulary
Expressive One-Word Vocabulary Test – 4 (EOWPVT – 4: Frauwirth et al., 2017) or Expressive One-Word Vocabulary Test Spanish Bilingual Edition (EOWPVT-SBE) Woodcock-Johnson IV Letter- Word Identification (Shrank et al., 2018) or Batería IV Woodcock- Muñoz – Identificación de letras y palabras (Batería IV; LaForte et al., 2019)
For students between 3-6-years-old, EOWPVT-4: = .94- .97; For students between 3-6-years-old, EOWPVT-SBE = .95-.96 For students between 3-6-years-old Batería IV – Identificación de letras y palabras =.97-.98
~10-15 minutes
Literacy
~5-7 minutes
Final Report: Pilot Study
11
Domain
Measure
Description
Reliability
Duration
Math
Woodcock Johnson-IV Applied Problems (Shrank et al., 2018) or Batería IV – Problemas Aplicados (LaForte et al., 2019)
Students’ math skills were measured using the Applied Problems or Problemas aplicados subtests of the Woodcock- Johnson IV and Batería IV, respectively. During this assessment students are presented with a testbook and asked to analyze and solve math problems. MEFS was used to measure students’ executive functioning skills (e.g., working memory, inhibition, and cognitive flexibility). MEFS is administered on a tablet or touch screen computer. Students are presented with cards that vary by animal and color. They are prompted to sort the cards based on one set of rules (e.g., by color) and then prompted to sort by a different set of rules (e.g., by animal). The Leiter-3 was used to assess students’ attention, activity level, mood regulation, organization skills, anxiety, sociability, impulse control, energy and feelings, and sensory reactivity during the assessment period. Two composite subscales were used, cognitive/Social and Emotions/Regulation. This is completed immediately after all other assessments by the trained assessor.
For students between 3-6-years-old Batería IV – Problemas aplicados =.91-.92
~5-7 minutes
Executive Function
Minnesota Executive Function Scale (MEFS; Beck et al., 2011)
ICC for students between the ages of 2.5 – 5 years-old is .93
~5-7 minutes
Social and Emotional Development
Leiter – 3 (Roid et al., 2013)
For students between 3-6-years-old Cognitive/Social = .96-.97 For students between 3-6-years-old Emotions/ Regulation = .91-.96
~15-20 minutes
Final Report: Implementation and Pilot Study
32
Procedures
All child level assessments were administered in the May of 2023. Students were assessed by trained research assistants on several measures of school readiness (e.g., math, literacy, executive function). If students spoke a language other than English at home (n = 18), they were screened using the Simon Says and Art Show subtests of the preLAS 2000 (Duncan & DeAvila, 1998). Scores on these two subtests were added to create a total score. If students met the cutoff (i.e., 15 out of 20), determined by a prior study (i.e., Rainelli et al., 2017), they received all assessments of school readiness in English. It is important to note, that though 14 out of 18 DLLs assessed on the preLAS 2000 scored higher than the 15-point English cutoff, they are still deemed DLLs as they were learning both English and Spanish. If students did not meet this cutoff (n = 4), they received all assessments in English and Spanish, except for the executive function and receptive vocabulary measures. The measure of executive function was only administered to these four students in Spanish. These four students were not assessed on the measure of receptive vocabulary. However, due to the small number of DLL students who fell below the cutoff, we opted to use English assessments in all analyses. Findings Below we describe the findings from our analyses of student assessments. We begin with a discussion of our descriptive analyses, which compares the students in the PKOMW classrooms and those in the comparison group. Next, we examined differences between the two groups on measures of school readiness. To do so, we first conducted t-tests to compare differences between the two groups on measures of school readiness without controlling for any variables. Then, we looked at the differences between the two groups, controlling for demographic and other variables (listed below). Lastly, we restricted the sample to only include dual language learners, and compared differences between the two groups of students, and controlled for demographics and contextual level variables. We end this section with a discussion of the measures where we observed the largest differences between the two student groups.
Descriptive Analyses
To understand whether students in the PKOMW classrooms and those in the comparison classrooms had similar backgrounds, we conducted a series of descriptive analyses. We examined differences between PKOMW and comparison group students on a range of demographic and classroom level characteristics. There were no significant differences in age (PKOMW: M = 57.62 months, SD = 4.02; Comparison: M = 54.83 months, SD = 7.64), sex, or ethnicity between students in the PKOMW and comparison groups. There was a marginally greater proportion ( p < .10) of DLLs in the comparison group than the PKOMW group (though both conditions had 9 DLLs participating in the study). See Table 1 for differences in demographic characteristics of PKOMW and comparison students. Finally, students in the PKOMW classrooms came from families with higher household income, had parents with higher levels of
Final Report: Implementation and Pilot Study
33
education, and were in classrooms with fewer students as compared to those in comparison classrooms.
Examining School Readiness Differences Between PKOMW and Comparison Students To better understand which demographic and contextual factors might impact students’ school readiness outcomes, we ran zero-order bivariate correlations. These correlations informed which variables were used as control variables in subsequent analyses. See Table 3 for Spearman’s correlations between dichotomous demographic and contextual variables and students’ school readiness outcomes. Being in the PKOMW group was moderately positively associated with receptive and expressive vocabulary and math skills. In other words, students in the PKOMW group scored higher on measures of receptive and expressive vocabulary and math skills. Ethnicity was also positively moderately associated with students’ emotions/regulations, where Hispanic students tended to score higher on measures of emotion regulation.
Table 3. Correlations Between Contextual and Demographic Characteristics and Students' Outcomes
Cognitive/ Social
Receptive Vocabulary
Expressive Vocabulary
Emotions/ Regulation
Executive Function
Literacy
Math
Treatment Group (PKOMW = 1)
0.58***
0.52**
0.08
0.36*
0.12
-0.15
0.01
Sex (Boy = 1)
-0.03
0.01
0.11
-0.003
-0.16
-0.14
-0.17
Ethnicity (Hispanic= 1)
-0.03
0.19
-0.31
0.03
0.13
.45**
0.09
DLL (DLL = 1)
-0.33
-0.13
-0.21
-0.19
-0.13
0.06
0.13
*significant difference, p < .05 **significant difference p < .01 ***significant difference, p < .001
Correlations results, as presented in Table 4, indicate significant positive associations between age and students' school readiness, suggesting that older children tend to exhibit higher levels of preparedness. Similarly, income and parents' educational level demonstrate positive correlations with child outcomes, indicating a potential link between socio-economic factors and school readiness. Furthermore, English language screener scores exhibit positive associations, suggesting that proficiency in English may be a contributing factor to enhanced school readiness among students, as the assessments that were analyzed were given in English. These correlations provide valuable insights into the relationships between demographic and contextual variables and children's preparedness for school. Specifically, findings showed older students generally scored higher in four of the seven assessed domains, expressive, math, cognitive/social and emotional regulation . Income was only associated with students’ receptive vocabulary; and p arents’ highest level of education was positively associated with receptive
Final Report: Implementation and Pilot Study
34
and expressive vocabulary, and literacy skills. Higher scores on the English language screener were positively associated with receptive vocabulary, literacy, cognitive/social, and emotion/regulation skills. Lastly, results showed that higher number of students in the classroom was associated with lower receptive vocabulary and lower cognitive/social skills. There were no other significant associations; thus, all other correlations are uninterpretable.
Table 4. Correlations Between Contextual and Demographic Characteristics and Students' Outcomes
Receptive Vocabulary
Expressive Vocabulary
Cognitive/ Social
Emotions/ Regulation
Executive Function
Literacy
Math
Age
0.20
0.44*
-0.08
0.36*
0.42*
0.39*
0.26
Income
0.39*
0.28
0.31
0.10
0.28
0.21
-0.24
Parent Highest Education
0.56***
0.39*
0.44*
0.24
0.09
-0.08
0.04
Years Teaching
-0.05
0.11
0.10
0.23
-0.21
-0.03
0.21
# of Students in Classroom English Language Screener
-0.43*
-0.24
-0.07
-0.02
-0.39*
-0.004
0.08
0.55*
-0.10
0.51*
0.41
0.68**
0.77***
0.45
*significant difference, p < .05 **significant difference p < .01 ***significant difference, p < .001
Initial Examination of Group Differences. Like analyses with students ’ demographic and classroom information, we sought to examine overall differences between the PKOMW and comparison, free of control variables. To do this we conducted a series of Independent Sample T-tests with students ’ school readiness domains (i.e., expressive, and receptive vocabulary, literacy, math, executive functioning, and social emotional skills) as outcomes. These analyses revealed that overall, students in the PKOMW group performed better in expressive and receptive vocabulary and math than students in the comparison group when not controlling for any other variables (Table 5). There were no significant differences in students’ scores on literacy, social-emotional, or executive functioning assessments. Some of the differences that are identified in these sets of analyses may be explained by some of the differences that were discussed above, under the descriptive analyses. For example, results in Table 5 show that students in the PKOMW group outperformed those in the comparison group across receptive and expressive vocabulary, as well as on measures of math. However, descriptive analyses revealed that students in the PKOMW group came from families with higher household income, had parents with higher levels of education and were in classrooms with fewer students as compared to those in the comparison group, variables which have been found to be associated with child outcomes (Kim et al., 2014).
Final Report: Implementation and Pilot Study
35
Table 5. Comparisons Between All PKOMW and Comparison Students
PKOMW Students (n = 21)
Comparison Students (n = 12)
M
SD
M
SD
Receptive Vocabulary
90.33**
19.34
73.08
6.96
Expressive Vocabulary
97.68*
16.09
84.18
4.83
Literacy
95.33
16.02
92.50
13.98
Math
90.15*
20.39
75.83
15.78
Executive Functions
94.43
11.22
95.83
8.21
Cognitive/Social
115.95
37.74
112.08
24.38
Emotions/Regulation
99.52
36.36
112.50
13.00
Note. This table depicts whole-group independent t-test analyses. No control variables were used in these analyses. *significantly higher, p < .05 ** significantly higher, p < .01 *** significantly higher, p < .001
Examination of Group Differences in the Overall Sample Accounting for Key Contextual Factors . In our descriptive analyses we found that the PKOMW and comparison students differed across several demographic and contextual factors. Students in the PKOMW classrooms had families with significantly higher household income, parents with higher levels of education, and fewer students in their classrooms than those in classrooms. These factors are often associated with higher scores on students’ school readiness assessments (Kim et al., 2014), as they were in the current study. To account for these factors, which could be driving the differences between PKOMW and comparison classrooms, we conducted Analysis of Covariance (ANCOVA) models that controlled for these variables. We also controlled for DLL status as it was marginally different across the two groups. See Table 6 for whole group analyses with control variables. When examining all students, controlling for DLL status, household income, pare nts’ highest level of education, and number of students in their classrooms there were no statistically significant differences between PKOMW and comparison groups on any assessments of school readiness.
Table 6. Comparisons Between All PKOMW and Comparison Students Accounting for Contextual Variables
PKOMW Students (n = 21)
Comparison Students (n = 12)
M
SD
M
SD
Receptive Vocabulary
90.30
18.85
73.73
6.91
Expressive Vocabulary
97.44
16.52
84.20
5.09
Literacy
95.70
16.35
93.45
14.24
Math
89.74
20.86
76.81
16.15
Executive Functions
94.45
11.51
95.55
8.55
Final Report: Implementation and Pilot Study
36
PKOMW Students (n = 21)
Comparison Students (n = 12)
Cognitive/Social
114.50
38.11
113.18
25.26
Emotions/Regulation 31.00 Note. This table shows whole-group ANCOVA analyses. There were no significant differences between PKOMW students and Comparison students ’ school readiness scores when controlling for DLL status, household income, parents’ highest level of education, and number of students in their classrooms. 98.35 36.90 103.03
Descriptive Analyses of Children Identified as DLLs
The PKOMW curriculum includes specific supports to facilitate learning and language development for DLL students. DLLs, or students who had a home language other than English were included in this set of analyses (whether they passed the English language screener or not). Due to this focus, we sought to determine if there were differences in scores between DLL students (n = 9) in the PKOMW classrooms and those in the comparison (n = 9) classrooms. Prior to conducting analyses, we examined differences between demographic variables, English screener, and number of students in the classroom across both conditions (see Table 7). These descriptive analyses show that a higher proportion of DLL students in PKOMW classrooms came from household with higher income, and had parents with higher levels of education, and were in classrooms with a smaller number of students, all of which have been found to be associated with child outcomes (Kim et al., 2014). We also found that DLL students in the comparison group ( M = 18.78, SD = 1.72) had marginally higher scores on the English screener than those in the PKOMW group ( M = 14.78, SD = 7.64).
Table 7. Descriptive Analyses of Children Identified as DLLs in PKOMW and Comparison Classrooms
PKOMW DLL Students (n = 9)
Comparison DLL Students (n = 9)
Overall DLL Sample (N = 18)
Sex
Boy
22.20%
56.60%
33.30%
Girl
77.80%
44.40%
66.70%
Household Income
Below $15,000
0
22.20%
11.10%
$15,000-$25, 000
0
22.20%
11.10%
$25, 001-$35, 000
0
11.10%
5.60%
$35, 001-$50,000
11.10%
44.40%
27.80%
$50,001-$65,000
0
11.10%
5.60%
$65,001 or more
44.40%
22.20%
33.30%
Parent Highest Education
8th grade or less
0
11.10%
5.60%
Final Report: Implementation and Pilot Study
37
PKOMW DLL Students (n = 9)
Comparison DLL Students (n = 9)
Overall DLL Sample (N = 18)
9th grade- 12th grade no diploma
11.10%
11.10%
11.10%
High school graduate or GED
33.30%
44.40%
38.90%
Some college but no degree
22.20%
22.20%
22.20%
Associates degree (AA or AS)
0
11.10%
5.60%
Bachelor’s degree (BA, BS, or AB)
11.10%
11.10%
11.10%
Graduate or professional degree
0
11.10%
5.60%
M
SD
M
SD
M
SD
Average number of students enrolled in class
17.10
6.56
23.67**
.49
19.48
1.06
English Screener
14.78
7.65
18.78
1.71
16.78
5.76
Note. Demographic information is missing for one child. † marginally higher, p <.10
*significant difference, p < .05 **significant difference p < .01 ***significant difference, p < .001
Examination of Group Differences in the DLL Sample Accounting for Key Contextual Factors .
For this set of ANCOVA analyses, we restricted the sample to only include DLL students in the PKOMW and comparison classrooms. In these analyses, we controlled for income, parents’ highest level of education, and number of students in classrooms. We also controlled for students ’ language screener score because the DLLs in this sample demonstrated a range of English language abilities, as demonstrated by their performance on the English screener. All analyses used students ’ English assessment scores, thus controlling for DLL students ’ English level helps partition out the variance in outcomes that can be attributed to language proficiency. In these analyses, we found that DLL students in the PKOMW classrooms performed higher in expressive and receptive vocabulary and math than DLLs in the comparison group, (See Table 8).
Table 8. Comparisons Between DLL PKOMW and Comparison Students Accounting for Contextual Variables
PKOMW DLLs (n = 9)
Comparison DLLs (n = 9)
M
SD
M
SD
Receptive Vocabulary
86.20***
19.76
72.00
4.07
Expressive Vocabulary
98.13*
9.93
83.13
3.83
Literacy
94.60
15.12
89.25
12.85
Final Report: Implementation and Pilot Study
38
PKOMW DLLs (n = 9)
Comparison DLLs (n = 9)
Math
91.22***
15.02
73.63
15.06
Executive Functions
97.10
9.48
93.63
7.50
Cognitive/Social
104.60
44.63
114.88
24.71
Emotions/Regulation 10.21 Note. This table shows DLL-specific ANCOVA analyses . These analyses controlled for DLL students ’ English screener scores, household income, parents’ highest level of education, and number of students in their classrooms. 98.30 37.42 115.00
† marginally higher, p <.10 *significantly higher, p < .05 **significantly higher, p < .01 ***significantly higher, p < .001
School Readiness Measures Exhibiting Greatest Between Group Differences
To determine which assessment demonstrated the biggest differences between PKOMW and comparison classroom students, mean differences (i.e., PKOMW mean minus comparison classroom mean) were examined. The greatest mean difference occurred in receptive vocabulary (Mean Difference = 17.25), followed by math (Mean Difference = 14.32), and expressive vocabulary (Mean Difference = 13.50). There were no significant differences between scores on literacy (Mean Difference = 2.83), Executive Function (Mean Difference = -1.40) cognitive/social skills (Mean Difference = 3.87), or emotions/regulation (Mean Difference = -12.98) (see Table 9).
Table 9. Mean Differences Between the PKOMW and Comparison Groups
PKOMW Students (n = 21)
Comparison Students (n = 12)
Mean Difference
Receptive Vocabulary
90.33
73.08
17.25**
Expressive Vocabulary
97.68
84.18
13.50*
Literacy
95.33
92.50
2.83
Math
90.15
75.83
14.32*
Executive Functions
94.43
95.83
-1.40
Cognitive/Social
115.95
112.08
3.87
Emotions/Regulation
99.52
112.50
12.98
Note. Analyses do not control for demographic characteristics. *significantly higher, p < .05 **significantly higher, p < .01 ***significantly higher, p < .001
Discussion The results showed that when comparing students in the PKOMW and the comparison group, when not controlling for demographic characteristics, PKOMW children scored higher on expressive and
Final Report: Implementation and Pilot Study
39
receptive vocabulary and math. However, when controlling for demographic characteristics, there were no statistically significant differences between the two groups on measures of school readiness. This study also found, that when examining only DLL students — that is, students who spoke a language other than English at home — PKOMW DLLs (n = 9) outperformed those in the comparison DLLs group (n = 9) on measures of expressive and receptive vocabulary and math. This was true even when controlling for key variables, such as household income, parents’ level of education , number of children in the classroom, and English language skills. The PKOMW curriculum targets students’ language, math, and literacy skills and aims to impact students executive functioning and social-emotional skills. Given PKOMW’s target domains , and specific focus on supporting DLLs, findings mostly align with expectations. Furthermore, the differential pattern in finding significant results for DLLs but not monolingual students, can be explained by prior literature. Previous studies have found that DLL students ’ second language vocabulary increases rapidly at school entry (Hammer et al., 2008; 2014). DLL students may receive a greater proportion of English language learning exposure in school; therefore, DLLs’ English vocabulary could be dependent upon classroom language experiences with peers and teachers. Conversely, monolingual English-speaking students enter classrooms hearing English at home and in school. Thus, their English vocabulary is less likely to depend solely on classroom input and therefore is more difficult, comparatively, to improve with classroom interventions alone. The finding that DLL students in the PKOMW classrooms performed better than those in the comparison classroom suggests that the DLL specific component of the curriculum could be particularly beneficial for DLL students. However, more research is needed (e.g., with a larger sample of students, DLLs with varying English skills) to further investigate the impacts of the PKOMW curriculum. There is also a need to better comprehend teacher preparation and professional development around working with DLLs, and the language background of teachers. These factors might also have played a role in how teachers used the PKOMW curriculum to support DLL children (Castro et al., 2017; Patika, 2023; Tang et al., 2012). Despite the strong literacy component in the PKOMW curriculum, student ’s literacy scores across the two groups did not differ after adjusting for demographic and other important contextual characteristics (i.e., controlling for DLL status, household income, parents’ highest level of education, and number of students in their classrooms). Additionally, there were no differences between students executive functioning and social-emotional skills. This could be due to several reasons. The PKOMW curriculum was intended for use with full-day 4-and 5-year-old preschool students; however, at the discretion of the participating district, it was implemented in half-day programs with three-and four-year-olds. All participating classrooms in this study were half-day. Teachers cited difficulties in using all the materials because they did not have enough time to enact all PKOMW activities. Additionally, teachers perceived that the curriculum was meant for older students. They reported that PKOMW was not developmentally appropriate for younger three-year old students. They discussed adapting the materials to meet the needs of students in their classroom. For example, some teachers cut down on the duration of book reading in each session, due to younger students ’ attention spans. These adaptations strayed from the intended implementation of PKOMW, potentially making it more difficult to detect an effect of the materials on students’ outcomes.
Final Report: Implementation and Pilot Study
40
There also is a need to understand the classroom contexts more broadly; however, the current study did not examine key contextual factors. This includes factors like classroom language use, support, and overall classroom quality. For example, studies demonstrate that use of Spanish in DLL classrooms is associated with greater math, language, reading, and social-emotional skills (Burchinal et al., 2012; Chang et al., 2007; Franco et al., 2019; Raikes et al., 2019). Within the context of the current study, teacher language ability and classroom instructional and non-instructional language use were not measured. Future studies examining the impacts of the PKOMW curriculum should investigate the efficacy of the curriculum with teachers who have varying levels of language proficiency or use differing levels of Spanish/English to determine if results replicate across a range of classroom contexts. Additionally, one teacher in the comparison classrooms stated that they used Guided Language Acquisition Deconstruction (GLAD) strategies to support language development for DLLs. These strategies could drive differences or make it difficult to detect differences in PKOMW and comparison classrooms. Therefore, future studies examining the impacts of PKOMW curriculum on students ’ outcomes should account for other strategies PKOMW or comparison teachers are using to facilitate DLL learning. In addition, the current study did not measure global classroom quality. While teachers were asked whether they used the curriculum, our team was unable to quantify what specific components of the curriculum were used and how well teachers delivered it. Classroom observation tools like the Classroom Assessment Scoring System (Pianta et al., 2008), detects aspects of overall classroom quality including Emotional Support, Classroom Organization, and Instructional Support that have been shown to be important predictors of students early learning outcomes across a large body of studies (Burchinal et al., 2012; Downer et al., 2012; Limlingan et al., 2020; White et al., 2020). Examining classroom quality in conjunction with fidelity to PKOMW implementation could be a critical piece for understating how PKOMW impacts monolingual English-speaking and DLL students alike. Finally, this study found that the greatest differences in scores across groups occurred in students ’ receptive vocabulary followed by math and expressive vocabulary. This aligns with findings in the implementation study, in which teachers appeared to favor materials that supported language (as opposed to literacy and math). A preponderance of evidence demonstrates that changes in receptive vocabulary, or the words that students understand, often precedes changes in expressive vocabulary, or the words that students use when speaking (Maier et al., 2016; Mancilla-Martinez et al., 2020). This is true for both monolingual and DLL students. These findings further demonstrate the need for support in enacting other areas of the curriculum but also demonstrate that PKOMW curriculum could be a great tool for supporting students ’ language development.
Conclusion The PreK On My Way ™ (PKOMW) pilot study has shed light on the challenges and successes associated with the implementation process and offer crucial recommendations for improving the curriculum's effectiveness.
Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Page 9 Page 10-11 Page 12-13 Page 14 Page 15 Page 16 Page 17 Page 18 Page 19 Page 20 Page 21 Page 22 Page 23 Page 24 Page 25 Page 26 Page 27Made with FlippingBook - Online catalogs