Research & Validation | Scholastic Literacy

Mixed Methods Evaluation of Scholastic Literacy in a South Carolina Public School District

Alan J. Reid, PhD Michael A. Cook, PhD Maria Jose Barros, PhD Steven M. Ross, PhD August 2023

EXECUTIVE SUMMARY

iii

Contents EXECUTIVE SUMMARY ............................................................................................. v Mixed Methods Evaluation of Scholastic Literacy in a South Carolina Public School District.................................................................................................................... v Research Design .................................................................................................. v Study Sample ......................................................................................................vi Program Impact on Reading Achievement..............................................................vi Teacher Program Perceptions .............................................................................. vii Conclusions ........................................................................................................ vii Scholastic Literacy................................................................................................ 1 Method ................................................................................................................... 2 Research Design .................................................................................................. 2 Participants.......................................................................................................... 2 Survey sample. ................................................................................................. 3 Measures............................................................................................................. 3 NWEA MAP Reading. ......................................................................................... 3 Teacher survey. ................................................................................................ 4 Teacher focus group. ........................................................................................ 4 Analytic Approach ................................................................................................ 4 Results ................................................................................................................... 5 Achievement analyses .......................................................................................... 5 Subgroup analyses............................................................................................ 7 Survey and Focus Group Results............................................................................... 8 Implementation ................................................................................................... 9 Perceived Impacts.............................................................................................. 12 Program Attitudes .............................................................................................. 14 Strengths ....................................................................................................... 14 Weaknesses ................................................................................................... 15 Recommendations.............................................................................................. 16 Discussion............................................................................................................. 17 Conclusions ....................................................................................................... 18 Appendix A: Baseline Equivalence Tables ................................................................ 20 Appendix B: Online Survey ..................................................................................... 21

EXECUTIVE SUMMARY

Appendix C: Focus Group Interview Protocol ........................................................... 33

EXECUTIVE SUMMARY

EXECUTIVE SUMMARY Mixed Methods Evaluation of Scholastic Literacy in a South Carolina Public School District

The impact of Scholastic Literacy on elementary reading achievement was the main quantitative focus of this mixed-methods study. The qualitative study portion broadly focused on the implementation of, and instructional practices used with Scholastic Literacy in a public school district in South Carolina, as well as teacher and student attitudes towards Scholastic Literacy, perceived impact on student engagement and achievement, and program support and professional development. Scholastic Literacy is a core literacy program that provides culturally relevant texts and targeted instruction in reading, writing, language, and word-study to support social-emotional development and help students become lifelong independent thinkers, readers, and writers. The program, designed for students in Grades K–6, is structured around six thematic units that clearly articulate a sequence of strategies and skills. There are three components: teacher-led whole-group instruction, teacher-led small- group instruction, and independent instruction. When combined, these instructional approaches utilize standards and data-informed differentiated strategies that target a range of reading and writing skills. Research Design During the 2022-23 school year, Scholastic Literacy contracted with the Center for Research and Reform in Education (CRRE) at Johns Hopkins University to compare the reading achievement of Scholastic Literacy students in a public school district in South Carolina to matched comparison student data obtained from a Similar Schools Report (SSR) prepared by NWEA. The SSR was used because all teachers in participating schools used Scholastic Literacy, meaning a within-district comparison group was not readily available. A Similar Schools Report contains data from students who, relative to the intervention (public school district) sample, come from schools in a similar area (urban, suburban, rural), with similar percentages of free and reduced meals students (FARMS), creating a “virtual comparison group” of students, and allowing for direct comparison of MAP score growth between students who used Scholastic Literacy and otherwise similar students who did not use Scholastic Literacy. The evaluation determined whether students who used Scholastic Literacy improved their reading more than those in comparison, controlling for prior achievement and other student covariates. The study also analyzed teacher attitudes toward Scholastic Literacy, perceived impact on student engagement and achievement, program support and professional development.

EXECUTIVE SUMMARY

Study Sample The present study sample included 2,665 Grades 2-5 students from across 12 elementary schools located in South Carolina. The student population consisted mainly of Black (34%) and Hispanic (32%) students, with a slightly smaller percentage of White (26%) students. The survey sample consisted of 44 teachers, while focus group teachers were drawn from three schools in South Carolina. Program Impact on Reading Achievement A small positive impact of Scholastic Literacy on student reading achievement was observed across the entire sample. The magnitude of this impact was 0.55 points but did not quite reach statistical significance ( p = .085). Significant positive impacts were observed for Grades 3 and 4 students, with Scholastic Literacy students in these grades outgaining their virtual comparison counterparts by nearly 1 point at each grade level. Figure 1 shows program impacts overall and at each grade level. No other significant positive program impacts were observed across other student subgroups of interest. This finding satisfies ESSA’s “promising” or Tier 3 evidence. Figure 1 Scholastic Literacy Impacts Overall and By Grade Level

Average MAP Growth Reading Gain Scores Overall and By Grade Level

15.92

15.81

11.24*

10.95^

10.4

10.33

9.07*

8.24

7.6

7.07

Cohort 1

Grade 2

Grade 3

Grade 4

Grade 5

Overall

Grade Level

Treatment

Comparison

Note , ^ p < .10; * p < .05.

EXECUTIVE SUMMARY

vii

Teacher Program Perceptions Teacher perceptions of Scholastic Literacy were generally positive, especially in regard to student engagement and achievement. Teachers implemented program components with varying degrees of fidelity; with components such as Read-Alouds, student resource books, and Shared/Close Reading Texts being used several times per week, while other resources such as Picture Cards, Writing Workshop Resource Books, Big Books, and Digital Family Guides were used less often or not at all by teachers. Teacher attitudes toward Scholastic Literacy were also generally positive, with especially positive attitudes observed regarding the structure and routine that it creates in the classroom presenting learners with highly engaging and authentic texts. Conclusions The key results and conclusions of this evaluation are as follows: • Scholastic Literacy students slightly outgained comparison students identified by NWEA’s Similar Schools Report, by 0.55 points. This advantage approached, but did not reach, statistical significance ( p = .085). When school-level clustering was not considered in analyses, this advantage was statistically significant ( p = .001). • Significant program impacts were evidenced in Grades 3 and 4, with students in these grades outgaining comparison students by nearly 1 point. No other significant program impacts were observed. • Teacher perceptions of Scholastic Literacy were generally favorable, especially in relation to student engagement and achievement. • Similarly, teachers reported overwhelmingly positive experiences of using Scholastic Literacy in their classrooms, with high program implementation fidelity generally being observed and reported. • Teacher attitudes toward Scholastic Literacy were also generally favorable, especially related to program elements promoting structure and routine in the classroom. • This study provides ESSA “Promising” or Tier 3 evidence of the efficacy of Scholastic Literacy in improving student achievement in reading for students in Grades 2-5.

SCHOLASTIC LITERACY

Mixed Methods Evaluation of Scholastic Literacy in a South Carolina Public School District

The impact of Scholastic Literacy on elementary reading achievement was the main quantitative focus of this mixed-methods study. The qualitative study portion broadly focused on the implementation of, and instructional practices used with Scholastic Literacy in select public schools in South Carolina, as well as teacher and student attitudes towards Scholastic Literacy, perceived impact on student engagement and achievement, and program support and professional development. Quantitative achievement data included NWEA MAP reading scores and demographic data from the school district, as well as reading achievement scores from a virtual control group (VCG), as provided by a NWEA Similar Schools Report (SSR). Qualitative methods and measures included both an online survey completed by teachers and in-person focus groups with 3 rd , 4 th , and 5 th grade teachers conducted at three different schools in South Carolina. Results of quantitative comparisons of reading achievement between students who used Scholastic Literacy and VCG students are presented in this report, as well as qualitative analysis relating to Implementation, Impact, and Program Attitudes. Additionally, recommendations for improving the implementation of Scholastic Literacy are included at the end of this report. This study was designed to address the following research questions: 1. To what degree did teachers feel prepared to implement Scholastic Literacy ( + F.I.R.S.T and W.O.R.D) with fidelity? 2. Do students receiving Scholastic Literacy (+ F.I.R.S.T. and W.O.R.D.) demonstrate statistically greater growth in reading skills when compared to a group that does not use the program? 3. Do changes in reading skills vary by student demographic characteristics such as gender, race/ethnicity, initial ability, free or reduced-price meals status, English learner status, or participation in special education? 4. What are the profiles of students’ usage of Scholastic Literacy (+ F.I.R.S.T and W.O.R.D) that are associated with changes in reading skills? 5. How does Scholastic Literacy impact student interest in reading? Scholastic Literacy Scholastic Literacy is a core literacy program that provides culturally relevant texts and targeted instruction in reading, writing, language, and word-study to support social-emotional development and help students become lifelong independent thinkers, readers, and writers. The program, designed for students in Grades K–6, is structured around six thematic units that clearly articulate a sequence of strategies and skills. There are three components:

SCHOLASTIC LITERACY

• Teacher-led whole-group instruction focuses on book-centered, standards- informed, and skills-rich reading and writing instruction. Teachers engage students using read-alouds, shared/close reading, cross-textual analysis, mentor texts, micro-lessons, and craft/structure analyses. • Teacher-led small-group instruction offers more data-informed differentiated instruction capitalizing on guided reading, strategy groups, phonics, foundational skills, word, study, and literature circles. • Independent instruction provides students additional learning support through three digital learning programs, F.I.R.S.T., W.O.R.D., and Literacy Pro. F.I.R.S.T. provides digital instruction in foundational skills (phonemic awareness, phonics, and comprehension) for students in preK-2; W.O.R.D. provides digital game- based vocabulary instruction built around 2,500-word families; Literacy Pro is a web-based program that provides access to a digital library (eBooks), as well as connections to a collection of hardcopy paperback titles that can be read offline. Through Literacy Pro, students can track the books they are reading, log their independent reading, and monitor their reading growth over time.

Method

Research Design This study analyzed reading progress monitoring assessments from the 2022-23 school year in the South Carolina school district. Specifically, Reading RIT Growth scores from NWEA’s Measures of Academic Progress (MAP) assessment were analyzed in these quantitative achievement analyses. A quasi-experimental design (QED) was used to compare reading achievement of Scholastic Literacy students to matched comparison student data obtained from a Similar Schools Report provided by NWEA, through district leaders. A Similar Schools Report contains data from students who, relative to the intervention (public school district) sample, come from schools in a similar area (urban, suburban, rural), with similar percentages of free and reduced meals students (FARMS). Additionally, students as a group are matched on the basis of grade level and prior MAP reading achievement, as well as demographic variables including gender and ethnicity. This creates a “virtual control group” of students, allowing for a direct comparison of MAP score growth between students who used Scholastic Literacy and otherwise similar students who did not use Scholastic Literacy. Participants A small-city school district of approximately 21,000 students across 32 schools located in South Carolina was used for this study. White students constitute the largest

SCHOLASTIC LITERACY

ethnicity group in the district (38%), followed by Hispanic/Latino) (30%) and Black (25.5%) students. Slightly more than 40% of students are FARMS-eligible, and approximately 17% of students are identified as ELLs. Demographics for Grades 2-5 students in schools using Scholastic Literacy in the 2022-23 school year are shown in

Table 1. Table 1 Student Characteristics of Analytic Sample Group

Percentages

% Black % White % Hispanic % Other Race % Female

34.67 26.15 31.78 7.39 49.83 68.89 14.93 22.63

% Economically disadvantaged

% Special Education

% ELLs

N 2,665 Slightly larger percentages of Black and Hispanic students are found in the analytic sample, in relation to district-wide demographics, while a slightly smaller percentage of White students was observed. Nearly 70% of students were classified as economically disadvantaged, well above the 40% rate across the district. Similarly, though to a small degree, the analytic sample contained a larger percentage of ELLs. Survey sample. The online survey was delivered by the Center for Research and Reform in Education (CRRE) to valid email addresses for district educators who are involved with the implementation of Scholastic Literacy. Survey participation varied in terms of demographics, teaching experience, and degree obtained, though the most common response originated from a Caucasian female teacher with multiple years’ experience teaching at the school where Scholastic Literacy is being implemented. Ten respondents (22.7%) indicated that this was their first-year teaching at that particular school, though this did not necessarily also mean that it was their first year of teaching altogether. There were at least three teacher respondents from every grade level (K-5), with the most taught grade level being 2 nd grade (25%, n = 11). Likewise, education levels varied, with half of the respondents (50%, n = 22) having earned a bachelor’s degree and 45.5% ( n = 20) having earned a master’s degree. Measures NWEA MAP Reading. NWEA MAP RIT Growth scores were obtained from the beginning, middle, and end of the 2022-23 school year. The participating school district

SCHOLASTIC LITERACY

provided NWEA MAP Reading assessment scores for all Grades 2-5 students in district elementary schools that used Scholastic Literacy. MAP RIT Growth scores are vertically scaled so that scores can be directly compared across grade levels. Table 2 shows ranges of MAP RIT Growth Reading scores for students in Scholastic Literacy schools at the end of the 2022-23 school year. Table 2 MAP RIT Reading Score Ranges, by Grade Grade MAP RIT reading score range Grade 2 147-238 Grade 3 147-231 Grade 4 144-248 Grade 5 143-249 Teacher survey. The online survey (Appendix B) was developed by researchers in the CRRE at Johns Hopkins University and delivered to teachers using the Qualtrics platform. The instrument focused on the implementation of Scholastic Literacy, along with perceived impacts and attitudes towards the program, and also included demographic items. Recommendations for improvement were also solicited from respondents. Teacher focus group. The focus group interview protocol (Appendix C) was also developed by researchers in the CRRE and focused on gathering more elaborative details on the implementation and experiences with Scholastic Literacy. A CRRE researcher conducted in-person focus groups with teachers at three different schools in South Carolina. Participant responses were analyzed and coded by CRRE research staff, and findings are presented thematically in this report. Analytic Approach Achievement data for students in Grades 2-5 were analyzed descriptively by examining patterns in NWEA MAP Reading scores. Virtual Control Group data obtained from NWEA were used as a comparison group to students who used Scholastic Literacy. The BOY MAP Growth Reading score was defined as the pretest measure, while the EOY MAP Growth score was defined as the posttest measure. When constructing the virtual comparison group for a Similar Schools Report, NWEA matches each student in the selected school district with multiple comparison students (as few as 3, up to as many as 51), on the basis of prior MAP Reading achievement and demographic variables. This process creates a “virtual comparison group” of students for each intervention student, allowing for a comparison of MAP score growth between students who used Scholastic Literacy and otherwise similar students who did not use Scholastic

SCHOLASTIC LITERACY

Literacy. The data included in the Similar Schools Report included MAP Reading scores from fall 2022 (BOY) and spring 2023 (EOY), as well as relevant summary statistics for the virtual comparison group. As students are matched with their virtual comparison counterparts and are otherwise similar in terms of prior achievement and demographics, dependent t-tests were conducted by CRRE to examine differences in MAP Reading growth between Scholastic Literacy students and virtual comparison students. Results Achievement analyses In this section, we describe the results of a quasi-experimental study (QED) comparing student MAP Reading test growth from fall 2022 to spring 2023 in the intervention school district with that of similar students who did not use Scholastic Literacy. As described in the methods section, each student who used Scholastic Literacy was matched with a set of comparison students from NWEA’s database of student scores. Comparison students were selected by NWEA on the basis of similarity on a number of variables, including prior MAP Reading achievement, type of school, FARMS status, and ethnicity. MAP Reading gain scores for treatment and virtual comparison students in Grades 2-5, both overall and by grade level and subgroup, are examined in these analyses. Baseline equivalence on MAP Reading scores is shown in Appendix A; as students were matched by NWEA on prior achievement, this requirement is essentially trivial, and baseline differences did not exceed 0.01 standard deviations on any grade-level comparison. We first descriptively examine MAP Reading achievement score trends for Grades 2-5 Scholastic Literacy students across the 2022-23 school year. It is important to note that these are scores for treatment students only in the selected public school district in South Carolina. We only included students with non-missing spring 2023 (EOY) and fall 2022 (BOY) scores in this analysis, as these two timepoints were of most interest across our achievement analyses. Table 3 displays MAP Reading score trends by grade across each test administration. Table 3 MAP Reading Scores, by Grade Grade Fall 2022 Winter 2023 Spring 2023 Fall-to-Spring Gain 2 nd ( n = 663) 165.20 174.39 181.00 15.81 3 rd ( n = 681) 179.46 186.26 190.70 11.24 4 th ( n = 689) 190.53 196.46 199.61 9.07 5 th ( n = 632) 199.34 204.76 206.94 7.60

SCHOLASTIC LITERACY

Fall-to-spring reading achievement gains were largest, on average, for Grade 2 students, with gains averaging nearly 16 points. Gains decreased as grade level increased, with average gains of 11 points observed for Grade 3 students, followed by 9-point gains for Grade 4 and less than 8-point gains for Grade 5 students. Next, we descriptively compare achievement gains by grade for Scholastic Literacy and comparison students identified in the Similar Schools Report. Table 4 shows average MAP Reading scores in fall 2022 and spring 2023 for each grade level. Table 4 Average MAP Reading Scores, by Grade, Fall 2022 to Spring 2023 Fall 2022 Spring 2023 Change Grade 2 (n = 663) Scholastic 165.20 181.00 15.81 Virtual Comparison 165.20 181.12 15.92 Grade 3 (n = 681) Scholastic 179.46 190.70 11.24 Virtual Comparison 179.40 189.73 10.33 Grade 4 (n = 689) Scholastic 190.53 199.61 9.07 Virtual Comparison 190.53 198.77 8.24 Grade 5 (n = 632) Scholastic 199.34 206.94 7.60 Virtual Comparison 199.28 206.35 7.07 Fall-to-spring gains for Scholastic Literacy students were generally comparable to or slightly larger than gains for virtual comparison students. Scholastic Literacy students outgained comparison students by slightly less than 1 point in Grades 3 and 4, while Grade 5 Scholastic students outgained comparison students by approximately one-half point. Virtual comparison students slightly outgained Scholastic students in Grade 2, but only by about one-tenth of a point. Next, we examine the impacts of Scholastic Literacy on MAP Reading growth gain scores, in relation to virtual comparison students, by conducting dependent (matched) t-tests on MAP Reading gain scores from fall 2022 to spring 2023. Table 5 shows estimated Scholastic Literacy impacts on MAP Reading gains by grade level, as well as across the entire sample. Students included in these analyses had non-missing fall 2022 and spring 2023 MAP Reading scores.

SCHOLASTIC LITERACY

Table 5 MAP Reading Gain Scores Relative to Virtual Comparison Students, Fall 2022 to Spring 2023 Grade level Estimate Standard Error p value* Grade 2 (n = 663) -0.11 0.375 .579 Grade 3 (n = 681) 0.92* 0.343 .033 Grade 4 (n = 689) 0.82* 0.328 .043 Grade 5 (n = 632) 0.54 0.327 .123 All students (n = 2,665) 0.55^ 0.172 .085 Notes. 1. ^ p < .10; * p < .05. 2. All p values are adjusted for school-level clustering. Across all students, Scholastic students averaged slightly more than 0.5 points larger score gains on the MAP Reading assessment than did virtual comparison students. This overall difference was significant at p < .10 but not at p <. 05. after taking school-level clustering into account 1 . When breaking down by grade level, Scholastic Literacy students in Grades 3 and 4 significantly outgained virtual comparison students, with slightly smaller than 1-point larger gains evidenced for Scholastic Literacy students at both grade levels. Grade 5 students outgained virtual comparison students by slightly more than 0.5 points, and virtual comparison students slightly outgained Scholastic Literacy students in Grade 2; however, neither of these differences reached statistical significance. Subgroup analyses. We also conducted subgroup analyses, where we compared gain scores of Scholastic Literacy students in subgroups of interest, in relation to virtual comparison students. The results of these analyses are summarized in Table 6. Table 6 MAP Reading Gain Scores Relative to Virtual Comparison Students, by Subgroup Subgroup Estimate Standard Error p value Female (n = 1,328) 0.38 0.237 .185 SPED (n = 398) -3.58*** 0.483 <.001 FARMS (n = 1,836) -0.02 0.206 .521 ELL (n = 602) -0.27 0.383 .688 Black (n = 924) -0.66^ 0.299 .082 Hispanic (n = 847) 0.09 0.308 .762 White (n = 697) 2.54*** 0.320 <.001 Other race (n = 197) 1.17^ 0.554 .065

1 When school-level clustering is not taken into account, this impact is statistically significant ( p = .001).

SCHOLASTIC LITERACY

Notes. 1. ^ p < .10; *** p < .001. 2. All p values are adjusted for school-level clustering. A significant positive impact of Scholastic Literacy was evidenced by White students, with an advantage of 2.5 points for Scholastic Literacy students in this subgroup. A significant negative impact was also found for special education students, although the sample size was fairly small, relative to the entire sample. No other significant program impacts were evidenced across student subgroups of interest in Scholastic Literacy students. An additional set of subgroup analyses examined the impacts of Scholastic Literacy on students with different levels of prior reading achievement. Specifically, we classified these students as having “low,” “medium,” or “high” levels of prior reading achievement. “Low” prior achievement was defined as having a fall 2022 MAP Reading Growth score below the 25 th percentile, while “medium” prior achievement was defined as having a fall 2022 MAP Reading growth score between the 25 th and 75 th percentile, and “high” prior achievement was defined as having a fall 2022 MAP Reading Growth score above the 75 th percentile. Note that these percentiles are based on national norms and are not specific to these intervention students. The results of these analyses are shown in Table 7. Table 7 MAP Reading Gain Scores Relative to Virtual Comparison Students, by Prior Achievement Percentile Estimate Standard Error p value* Low ( n = 1,050 ) -0.029 0.317 .928 Mid ( n = 1,119 ) 0.718* 0.250 .045 High ( n = 496 ) 1.395*** 0.289 <.001 Notes. 1. * p < .05; *** p < .001. 2. All p values are adjusted for school-level clustering Significant positive program impacts were observed for students with medium and high prior reading achievement, with medium prior achievement treatment students outscoring virtual comparison students by more than 0.7 points, and high prior achievement students outscoring virtual comparison students by nearly 1.4 points. No significant program impacts were observed for students with low prior reading achievement. Survey and Focus Group Results Survey respondents ( n = 44) consisted of Scholastic teachers familiar with the implementation of Scholastic Literacy. Voluntary participants responded to the online survey and the focus group interview regarding (a) their background and demographics; (b) their experiences with implementing Scholastic Literacy; (c) their

SCHOLASTIC LITERACY

perceptions regarding the program’s impact on students; and (d) their attitudes towards the program, including recommendations for improving Scholastic Literacy. Implementation Prior to its implementation, teachers received professional development on Scholastic Literacy, though this varied by school. For most, the initial professional development sessions consisted of a full day of training prior to the start of school, and there were routine check-ins from Scholastic representatives throughout the school year. The initial training was viewed as being informative, but teachers felt that it could have been improved through more “small group instruction” and “seeing the lessons being modeled.” Survey respondents requested more training on the writing aspect of the program, specifically regarding how to use graphic organizers more effectively and how to manage their class time in order to address all of the components. One focus group participant suggested that Scholastic “embed discussions within the training about how to structure and plan” for implementation. Overall, 62.8% ( n = 27) agreed that the group training on Scholastic Literacy adequately prepared them for implementation; a similar number (58.1%, n = 25) indicated agreement with the statement “I am pleased with the Scholastic Literacy professional development I received.” Survey respondents were asked to indicate the frequency with which they engaged with each of the various components of Scholastic Literacy. Teachers reported that some of the components were never used; for instance, half of teacher respondents reported never using the Digital Family Guides, and 63.6% of respondents said they never used the Big Books. More than one-third of respondents never used the Whole-Class Flip Charts, Picture Cards, Authentic Texts with Teaching Cards, or the Writing Workshop Resource Books. Conversely, some of the components were used regularly and with fidelity, as displayed in Figure 2. Most predominantly, the Teacher’s Edition and the Read-Aloud features were central to instruction for most teachers.

SCHOLASTIC LITERACY

Figure 2 Frequency of Use for Scholastic Literacy Components

Scholastic Literacy Components

Teacher's Edition

95.5%

Read-Alouds

95.5%

Student Resource Books

6.8%

86.4%

Shared/Close Reading Texts

6.8%

84.1%

Strategy Posters and Anchor Charts

9.1%

25.0%

56.8%

Leveled Book Room

15.9%

18.2%

20.5%

45.5%

Mentor Texts

25.0%

15.9%

18.2%

40.9%

Teacher's Hub/Shared Close Reading Digital Library

13.6%

22.7%

25.0%

38.6%

Whole-Class Flip Charts

36.4%

13.6%

36.4%

Implementation Guides

11.4%

34.1%

22.7%

31.8%

Authentic Texts with Teaching Cards

38.6%

15.9%

20.5%

25.0%

Picture Cards

36.4%

22.7%

15.9%

25.0%

Writing Workshop Resource Books

38.6%

20.5%

Big Books

63.6%

11.4%

9.1%

15.9%

Digital Family Guides

50.0%

31.8%

9.1%

Never

Rarely About once per week Several times per week

Note. + < 5%.

The majority of teacher respondents indicated the highest popularity for the following Scholastic Literacy features, which were used at least several times per week: Teachers’ Edition (95.5%, n = 42), Read-Alouds (95.5%, n = 42), Student Resource Books (86.4%, n = 38), and Shared/Close Reading Texts (84.1%, n = 37).

SCHOLASTIC LITERACY

In addition to identifying the most frequently used components of Scholastic Literacy, teachers were asked to classify specific instructional activities as being either a central part, small part, or not a part of their reading instruction (see Table 8). Table 8 Instructional Activities by Order of Importance

Central to my reading instruction

Small part of my reading instruction

Not part of my reading instruction 0.00% (0) 2.27% (1)

93.18% (41) 6.82% (3) 90.91% (40) 6.82% (3)

Work with small groups of students. Teach whole class reading lessons.

Students apply pre-reading strategies such as previewing the text, accessing prior knowledge, formulating questions, clarifying understanding, setting a purpose, and making predictions. Students apply during-reading strategies, including visualizing, making connections, monitoring understanding, making logical inferences from details, rereading, questioning, and summarizing. Students apply after-reading strategies, including comparing, synthesizing, and drawing conclusions. I teach word-learning strategies, including use of word parts (roots, prefixes, and suffixes), context, and the dictionary. Work one-on-one with students on reading. I teach specific strategies for decoding unfamiliar words. I teach decoding/phonics skills while reading stories. Use tests to determine progress on skills. Students isolate or categorize sounds in words, segment words into sounds, blend sounds to form words, add sounds to words, delete sounds from words, and manipulate onsets and rimes. I provide opportunities to play with sounds through activities involving pictures, letter tiles, rhyming, and music to develop student

88.64% (39) 9.09% (4)

2.27% (1)

84.09% (37) 13.64% (6)

2.27% (1)

79.55% (35) 18.18% (8)

2.27% (1)

75.00% (33) 18.18% (8)

6.82% (3)

72.73% (32) 20.45% (9)

6.82% (3)

72.73% (32) 22.73% (10) 4.55% (2)

68.18% (30) 27.27% (13) 4.55% (1) 68.18% (30) 29.55% (12) 2.27% (2)

59.09% (26) 34.09% (15) 6.82% (3)

52.27% (23) 31.82% (14) 15.91% (7)

SCHOLASTIC LITERACY

awareness of the sounds in the English language.

Most commonly, teachers used a combination of working with small groups and with the whole class for their reading instruction. Central to this instruction were pre-reading (88.6%, n = 39), during-reading (84.1%, n = 37), and after-reading (79.6%, n = 35) strategies that were outlined in Scholastic Literacy. Nearly half of teachers indicated that the program has influenced the way they teach reading (47.7%, n = 21). Teacher participants in the focus groups were asked about the challenges they faced while implementing Scholastic Literacy. A common response to this question was that the program becomes easier with more use. According to one teacher, “It takes a lot of time to learn. It’s not one you can just pick up and go. You have to set routines. It takes a while.” Importantly, the theme of “time” came up often in teacher responses and can be summed up with one teacher’s comment: “Time. It’s always time. We don’t have it.” Especially for teachers implementing Scholastic Literacy in their first year, time management is difficult because of the amount of content to be covered. A focus group participant described this as being “stressful,” and another teacher compared the initial implementation to “fitting 20lbs into a 10lb bag” because of the time constraints that all teachers across all grades experience. As a result, teachers admitted that it is often the writing piece that is left out at the end of the lesson, due to a lack of time. In some cases, teachers found themselves having to “adapt the writing assessment” to be able to fit it into the fifth day of the lesson. Teachers benefited the most when they were provided with instructional materials earlier in the summer (as opposed to last-minute, or even after the start of the school year), allowing them more time to prepare. As noted by several respondents, implementation becomes easier over time as teachers find ways to “make it their own.” Perceived Impacts Teachers gave the overwhelming sense that Scholastic Literacy positively impacts their students in two ways, specifically: achievement and engagement. Regarding achievement, respondents credited Scholastic Literacy with quantitative gains on formal assessments (“I had 250% growth on MAP.”) Regarding specific skills (see Figure 3), teachers agreed that the program has helped improve students’ understanding of print concepts (59.2%, n = 26), and reading comprehension (66%, n = 29). However, there was less support for other impacts of Scholastic Literacy. Teachers disagreed that the program improved student writing (43.2%, n = 19) or their phonological awareness (36.3%, n = 16).

SCHOLASTIC LITERACY

Figure 3 Perceived Impacts on Student Skills

Impacts on Students' Skills

Scholastic Literacy has helped to improve my students' understanding of print concepts.

6.8%

15.9%

18.2%

29.6%

Scholastic Literacy has helped to improve my students' reading comprehension.

6.8%

11.4%

15.9%

36.4%

29.6%

Scholastic Literacy has helped to improve my students' phonics and word recognition.

22.7%

9.1%

29.6%

27.3%

11.4%

Scholastic Literacy has helped to improve my students' phonological awareness.

22.7%

13.6%

27.3%

9.1%

Scholastic Literacy has helped to improve my students' writing.

29.6%

13.6%

18.2%

31.8%

6.8%

Strongly disagree Somewhat disagree Neither agree nor disagree Somewhat agree Strongly agree

There were other discernable measures of student growth. For instance, teachers noticed a change in students’ maturity levels (“They grew so much—the way they were thinking and talking”) and in their confidence levels (“They’re more confident in reading. They don’t see it as a chore anymore.”) Vocabulary development was yet another area of noticeable growth in students, as supported by these teacher comments: [Students’] academic vocabulary has grown a lot. They are starting to use some of the words from the discussions.

SCHOLASTIC LITERACY

I believe the vocabulary lesson and the students' required use of vocabulary really had a major impact on student achievement. These improvements may also be tied to a perceived increase in student engagement. Teachers reported high levels of student interest and engagement when using Scholastic Literacy. Most teachers (61.4%, n = 27) agreed with the statement that “Students were engaged in Scholastic Literacy.” This is attributed to how the program “invites students to the discussion” and “sparks their thinking.” Additionally, teachers felt that Scholastic Literacy has helped students persevere when learning new concepts (54.5%, n = 24). Summarily, a focus group teacher was quoted as saying, “The growth makes you speechless.” Program Attitudes Teachers were asked to identify strengths and weaknesses of the Scholastic Literacy program, and responses were plentiful. Although teachers had different experiences with Scholastic Literacy, this depended on their school, support network, and personal motivation to implement the program with fidelity, so this section will only discuss the emergent themes in the collective responses and not list every single quality of the program that was given as a strength or weakness. Naturally, the question, “What do you like best about Scholastic Literacy? Least?” invites responses with wide- ranging criticism and praise for the program, some of which are highly individual and anecdotal. The following section presents the most common patterns and emergent themes in teacher responses. 2 Strengths The most commonly identified strength of the program was the diverse range of highly engaging and “excellent” texts and the Read-Aloud activities that are structured around them. Teacher respondents on the survey and in focus groups appreciated the quality of texts, with one teacher saying, “I mean, it’s Scholastic . Of course, their texts are good.” Specifically, teachers liked the “authenticity” of the texts and the deep selection of genres and stories, which made them “not feel like workbook texts.” Programmatically, teachers like the predictability that Scholastic Literacy creates in the classroom. The structured program helps students to “know exactly what is going to happen” and “takes the stress off of teachers.” Other teachers agreed that the

2 Additionally, a separate survey was administered to 64 teachers in the same district by Scholastic, and the results largely concurred with the findings from the CRRE survey. The strongest components of Scholastic Literacy, according to teachers, were the diversity of texts (62.9%, n = 39), the informational texts (59.7%, n = 37), literacy selections (58.1%, n = 36), and vocabulary development (50%, n = 31). The weakest components were identified as writing instruction (85.2%, n = 52) and phonics instruction (59%, n = 36).

SCHOLASTIC LITERACY

predictability was a strength of the program—both for teachers and students. Some related comments included: [Students] learned what to expect and know what that looks and feels like. I enjoy that it provides me with the opportunity to dig deeper as well as the many activities that it provides me. It was so easy to step into this curriculum. I enjoy the reading strategies and how we implement them in the read alouds, as well as how well the graphic organizers blend with the text we read. I like the detailed plans and strategies in place for the teachers to implement. Overall, teachers found Scholastic Literacy to be valuable in shaping the classroom routine—creating expectations for student learning and helping teachers to organize and implement their lessons with structure. Weaknesses Survey and focus group respondents highlighted areas of Scholastic Literacy that they felt were weak. With regard to the program structure, several respondents felt that there is too much “passivity” and not enough active learning taking place. This can be summed up in one teacher’s comment: “There needs to be more of an ‘I think/do,’ ‘we do,’ ‘you do’ approach. Teachers are doing all of the talking and students are passive learners.” This is more of a general comment than a specific pinpointing of a program weakness, but it was echoed by others. Perhaps the most commonly identified weakness of Scholastic Literacy has to do with the graphic organizers. For many teachers, there was a lot of confusion for how to use the graphic organizers effectively, especially for lower grades. According to one teacher: “I hate the graphic organizers for the little ones … they are so far from developmentally appropriate. … My students are 5 years old and not able to sit down to complete [it].” Other grievances with the graphic organizer included: [Students] don’t like doing the graphic organizer on their own. Some kids seem to get it—for others, it seems to be a struggle. That’s where the group work comes in. Sometimes skill doesn’t align with the graphic organizer. The graphic organizers are too repetitive and boring every week.

SCHOLASTIC LITERACY

A commonly expressed sentiment was that the graphic organizers were useful for organizing ideas, but “there is a step missing … putting it all together.” In other words, students lacked the ability to make the leap from using the graphic organizer to crafting a piece of writing. There needs to be explicit instruction on how graphic organizers should inform writing practice. One teacher found the graphic organizer to be a valuable tool for one-on-one conferencing: “Once I started conferencing with the organizer they did, that helped.” The fact that students do not have access to the texts at all times was a frequent area of concern for many teachers. During the Read-Alouds, students do not have the text in-hand. This is problematic for many teachers, who commented: Love for them to be able to revisit the text in their hands, but its only projectable. To have a book in their hands would be so nice. They read so much on the screens. It’s hard to analyze a text unless you can re-read it. They’ve only listened to it. They can’t go back and re-read. KIDS NEED A COPY OF ALL TEXTS. I cannot say this enough. They need it for attention purposes. Even if this is provided digitally. Another area of weakness, as identified in survey and focus group responses, was with the Scholastic Literacy writing assessments. Multiple different issues emerged with writing, though some themes centered on the lack of formative writing assessments throughout the unit, a “lack of scaffolded support,” and misalignment between the reading and writing assignments. Some teachers expressed that they had to improvise in some areas—for example, using the “Think Mores” as formative assessments and modifying the standard writing rubrics that were provided by Scholastic. Recommendations Teacher respondents gave their personal recommendations for how to improve Scholastic Literacy. Some of the open-ended responses were extremely granular and specific only to one teacher’s experiences with the program. Therefore, the recommendations included in this section reflect themes that emerged from multiple teachers’ perspectives.

SCHOLASTIC LITERACY

• Provide texts for all learners (either in print or digital formats, though teachers expressed a preference for print). This allows students to engage more with the text and practice reading strategies like previewing and re- reading for comprehension and understanding. • Build in short lessons for remediation on assumed prior knowledge . Optional activities could be embedded into the lesson to ensure foundational knowledge of the topic (e.g., grammar, figurative language, writing mechanics, and text features). One teacher suggested a short lesson that pulls a sentence from a Read-Aloud and diagrams it to teach parts of speech. • Include a variety of formative writing assessments . Teachers recommended creating shortened, condensed writing assessments that are introduced throughout the lesson to formatively assess student writing. Additionally, the assessment should reflect the writing technique being taught rather than in a multiple-choice format. • Align reading and writing . Teachers pointed out that in several cases, the reading genre did not match the writing activity (e.g., an informational text might ask students for a piece of fiction writing). This can be confusing for learners. Discussion The current study was a mixed-methods evaluation designed to provide efficacy evidence for the Scholastic Literacy program, as well as to provide data regarding program implementation and teacher perceptions of the program. Impacts on student reading achievement for Grades 2-5 students were determined by comparing treatment students who used the program to comparison students identified by NWEA’s Similar Schools Report who did not use the program. Results of the main impact analyses showed a small positive impact of Scholastic Literacy on reading achievement. Treatment students who used Scholastic Literacy averaged slightly more than half-point larger gains on the NWEA MAP Reading assessment from BOY to EOY, in relation to comparison students, although this impact did not quite reach statistical significance ( p = .085). It is important to note that, when school-level clustering was not taken into account, the main Scholastic Literacy impact on MAP Reading score gains was statistically significant ( p = .001). Significant positive impacts were evidenced for Grades 3 and 4 students, with Scholastic Literacy students in these grades averaging nearly 1-point larger MAP Reading gains than did comparison students. Subgroup analyses did not show any significant positive program impacts for student subgroups of interest.

SCHOLASTIC LITERACY

Survey respondents and focus group participants had an array of experiences with Scholastic Literacy in their classrooms, most of which were overwhelmingly positive. Regarding the usage of Scholastic Literacy, teacher respondents implemented certain aspects of the program routinely and with high fidelity. Namely, teachers incorporated Read-Alouds (and the Teacher’s Edition), student resource books, and Shared/Close Reading Texts several times per week, where they encouraged pre-, during-, and after-reading strategies in small groups and in whole class reading lessons. These activities would become central to most teachers’ weekly and even daily instructional practices. These resources and activities were identified as the greatest strengths of the program. Naturally, some of the components of Scholastic Literacy were not utilized as often or at all—e.g., Picture Cards, Writing Workshop Resource Books, Big Books, or Digital Family Guides—though this also may be attributed to the time constraints of teaching and is not necessarily a reflection on the quality of the materials. Similarly, some instructional activities were under-utilized, including opportunities to play with sounds such as rhyming and music to develop awareness of sounds in the English language. Teachers perceived Scholastic Literacy as being highly impactful on learner engagement and achievement. Responses on the survey and in focus groups provided evidence that the majority of teachers view Scholastic Literacy as a positive force in student engagement and in their achievement, specifically with the understanding of print concepts and reading comprehension. There was less of a perceived impact on phonics, word recognition, and writing. Focus group participants reinforced these findings, identifying the writing assessments as a shortcoming of the program. Attitudinally, teachers reacted favorably to Scholastic Literacy, particularly regarding the structure and routine that it creates in the classroom presenting learners with highly engaging and authentic texts. There are some areas of improvement, as noted earlier, which mostly centered around the accessibility to print texts, content alignment, and the design of more robust formative and summative writing assessments. Overall, teachers responded positively to Scholastic Literacy and its impact on learners. For teachers of all backgrounds and experience levels, the program provides an infrastructure for the reading classroom that becomes easier to implement each year. Conclusions The key results and conclusions of this evaluation are as follows: • Scholastic Literacy students slightly outgained comparison students identified by NWEA’s Similar Schools Report, by 0.55 points. This advantage approached, but did not reach, statistical significance ( p = .085). When school-level clustering was not considered in analyses, this advantage was statistically significant ( p = .001).

Page i Page ii Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Page 9 Page 10 Page 11 Page 12 Page 13 Page 14 Page 15 Page 16 Page 17 Page 18 Page 19 Page 20 Page 21 Page 22 Page 23 Page 24 Page 25 Page 26 Page 27 Page 28 Page 29 Page 30 Page 31 Page 32 Page 33 Page 34 Page 35 Page 36 Page 37 Page 38 Page 39

Made with FlippingBook - Online catalogs