Research & Validation | RISE & RISEUp EOY Results

Scholastic RISE TM andRISE TM Up MOY-EOY, 2022-2023 Report (Grades 1-5) Star Early Literacy, Star Reading, F.A.S.T., and FastBridge Research led by Rachel L. Schechter, Ph.D. Learning Experience Design (LXD) Research Analysis conducted by Paul A. Chase, Ph.D. and Lynch Research Associates’ Alicia D. Lynch, Ph.D. Contributions by Isabella Ilievski and Destiny Riley from LXD Research

Executive Summary

Overview Scholastic Inc. partnered with LXD Research to conduct a third-party evaluation of the Scholastic RISE and RISE Up literacy intervention program as it was implemented in a Florida school district during the 2022-2023 school year. The Scholastic RISE and RISE Up program is a Tier 2 intervention program designed for students in grades 1-5 that provides targeted small-group instruction to address and prevent reading gaps using explicit, structured comprehension, word study, phonics instruction, and guided writing for 45-60 minutes per day. The treatment program features structured literacy components to focus on phonemic awareness and other skills, such as writing and synthesizing knowledge, which difers from a typical core reading program. In this study, all elementary schools used Benchmark Advance as a core reading curriculum and the comparison schools used School Specialty’s SPIRE or Fountas and Pinnell’s Leveled Literacy Intervention. The study utilized a mixed-methods approach, including a matched quasi-experimental design complemented by classroom observations, teacher surveys, and interviews with school leaders and interventionists. The study used a variety of assessment tools and metrics of quantitative data, including Renaissance Star Reading, Cambium F.A.S.T., and FastBridge progress monitoring, along with student demographic data and program implementation logs flled out by the teachers each week. After Propensity Score Matching, data were collected from 19 treatment schools totaling 168 students and seven comparison schools totaling 252 students. This report describes the study methodology, fndings, and conclusions and provides recommendations for product improvement.

Key Findings

Quantitative Results Overall Improvement: There was a notable improvement in students' literacy scores and benchmark levels from winter 2022 to spring 2023. Both the RISE and comparison groups exhibited strong growth. All grades (1-5) demonstrated improvements in their benchmark percentages. Comparative Analysis with Non-RISE Students: Compared with students not exposed to RISE, the growth patterns were similar in Star and FAST assessments for grades 1-3 and 3-5 combined. Given its inaugural year, the ease of RISE integration points to its potential utility, especially with students previously resistant to other intervention tools. Even with completing only one cycle of four stations a week, there was a positive correlation between lesson number and outcomes. In other words, completing a higher level in the program was associated with higher scores. While RISE students in frst and second grade had higher gain scores than the comparison group, the sample size was not large enough for this diference to reach statistical signifcance. FastBridge Performance: A segment of the treatment and comparison school students took the FastBridge exam every two weeks during the second half of the academic year. While both groups exhibited positive trajectories, the RISE students demonstrated signifcantly larger growth from January 2023 to May 2023. Predictive Nature of the Final Lesson Number: Among grade 3-5 students, the fnal lesson number, indicated by the teacher implementation logs, was a signifcant predictor of the FAST EOY Scale Score after accounting for the BOY scale score.

Challenges: It is pertinent to note that the study had its challenges. The sample size was relatively small, and some external factors delayed the initiation of RISE implementation.

Qualitative Insights and Feedback Teachers and interventionists commended the quality of RISE materials and their efect on student engagement. Some commonly repeated positives focused on the high quality of RISE materials, which included the books, passages, and text-based questions; the completeness of the program; and the high-interest component of the subjects included in the readings. During in-person observations, educators exclaimed their appreciation for the material organization, the fow of daily lessons, and the feeling that the material was challenging enough for their students. Teachers

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

frequently mentioned the high quality of the print materials and praised Scholastic’s dedication to providing high-interest texts in diferent genres and the usage rate of the trade books and the Short Read Cards supported these claims. However, the qualitative analysis revealed potential areas for program enhancement and future studies. Recommendations Considering the results of this study, the following suggestions are recommended. Detailed suggestions for product enhancements (Appendix 9). Enhance RISE Materials: Primary recommendations included: ● Inclusion of visual support materials. ● Guidance on transitioning from graphic organizers. ● Enhanced emphasis on comprehension-centric academic language. ● Integration of daily progress monitoring tools. ● Facilitation of online data entry. Re-evaluate Implementation Strategy: Scholastic recommends using RISE in a station-rotation model with one to four instructors for 45-60 minutes daily. They told teachers they could use the program for 30 minutes and one teacher, for that is what the district’s intervention model would allow. Data collected by a portion of teachers suggested that the students used the program for about half of the shortened recommended time (60-75 minutes a week). Comparison Groups: District leaders were unable to report to the researchers what intervention products students in the comparison groups used. To support product evaluation, digitally tracking student intervention along with their monitoring of skill achievement will be helpful for decision-making. Expand Research Dimensions: Future research should consider evaluating student writing outcomes, given that the program materials say that RISE improves comprehension, word-solving, fuency, and writing. Researchers observed students highly engaged in writing activities and teachers reporting improvements in students' writing skills during the study period. In conclusion, while the initial results are promising after one year of use, continued research and enhanced methodologies will be critical to understanding and leveraging the full potential of RISE and RISE Up in improving student literacy among students grades 1-5.

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Tableof Contents E XECUTIVE S UMMARY ......................................................................................................................................... 1 T ABLE OF F IGURES AND T ABLES .......................................................................................................................... 5 I NTRODUCTION .................................................................................................................................................. 8 O VERVIEW OF THE P ROGRAM ..............................................................................................................................8 T REATMENT G ROUP :P ROGRAM K EY F EATURES ......................................................................................................9 C ORE R EADING P ROGRAM ................................................................................................................................... 10 R ESEARCH Q UESTIONS ..................................................................................................................................... 10 R ESEARCH D ESIGN ........................................................................................................................................... 10 S AMPLE ............................................................................................................................................................ 11 M ATCHING P ROCEDURES ...................................................................................................................................... 12 G ROUP L ITERACY S CORES B EFOREAND A FTER PSM FOR S TARAND FAST......................................................... 16 B ENCHMARK S TATUS FOR M ATCHED G ROUPS .......................................................................................................18 S ITE V ISIT S AMPLE ........................................................................................................................................... 19 D ATAAND M EASURES ...................................................................................................................................... 19 D ESCRIPTION OF A CHIEVEMENT D ATA ................................................................................................................. 19 D ESCRIPTION OF L OG D ATA ................................................................................................................................. 22 D ESCRIPTION OF T EACHER S URVEYS ..................................................................................................................... 22 D ESCRIPTION OF P RINCIPAL I NTERVIEWS ...............................................................................................................22 D ESCRIPTION OF T EACHER O BSERVATIONS ............................................................................................................22 A NALYTICAL A PPROACH ...................................................................................................................................23 T IME P ERIODOF A NALYSIS ....................................................................................................................................23 C ORRELATION BETWEEN L ESSON A CHIEVEMENT AND L ITERACY S CORES .............................................................. 23 F AST B RIDGE A NALYTIC A PPROACH ...................................................................................................................... 23 D ESCRIPTION OF HLM AND ANY COVARIATES USED IN THE MODEL .......................................................................25 Q UANTITATIVE F INDINGS ..................................................................................................................................26 L OG C OLLECTION D ESCRIPTION ...........................................................................................................................26 I NSIGHTS FROM L OG D ATA ................................................................................................................................... 28 B IWEEKLY F AST B RIDGE R EPEATED M EASURES ANOVAF INDINGS ....................................................................... 30 F IRST G RADE STARR ESULTS .............................................................................................................................. 32 S ECOND G RADE STARR EADING R ESULTS .......................................................................................................... 32 G RADES 3 THROUGH 5FASTR ESULTS .................................................................................................................32 RISES TUDENTS ’L ITERACY A CHIEVEMENT .......................................................................................................... 34 T EACHER S URVEY F INDINGS ............................................................................................................................. 39 S ITE V ISIT F INDINGS ........................................................................................................................................ 42 P ROGRAM R ECEPTION AND R ECOMMENDATIONS FOR I MPROVEMENT .................................................................43 C ONCLUSION ....................................................................................................................................................45

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

R EFERENCES .....................................................................................................................................................47 S TAFFING N OTES ................................................................................................................................................... 48 A PPENDICES ..................................................................................................................................................... 49 A PPENDIX 1: F AST B RIDGE COUNT AND PERCENTAGE OF PARTICIPANT GROUP BY GRADE ........................................ 49 A PPENDIX 2: S TAR R ESULTS .................................................................................................................................. 51 A PPENDIX 3:G RADES 3-5R ESULTS ....................................................................................................................... 54 A PPENDIX 4:G RADE 3O NLY ................................................................................................................................58 A PPENDIX 5:T- TESTS FOR BOY AND MOYL ITERACY S CORES BY G ROUP S TATUS ................................................62 A PPENDIX 6.RISEE DUCATOR S URVEY R ESULTS .................................................................................................. 66 A PPENDIX 7.O BSERVATION P ROTOCOL A NALYSIS ................................................................................................. 81 A PPENDIX 8. S ITE V ISIT M EMO ,O BSERVATION I NSIGHT R EPORT S UMMARY ..........................................................90 A PPENDIX 9.RISE/U P I NTERIM I NSIGHT R EPORT ,W INTER 2023...................................................................... 91

Table of Figures and Tables Table 1. Structured Literacy Characteristics in RISE and RISE Up Lessons Table 2a. Number of Interventionists are in the Treatment and Comparison Groups Table 2b. Number of Students and Schools per Grade and Condition Table 3. Number of Students per Grade and Condition Before and After PSM

10 11

13 Table 4a. PRE-PSM percent of Students Male, Ethnic Minority, FRL, ELL, and SPED by Grade and Condition 13 Table 4b. POST-PSM percent of Students Male, Ethnic Minority, FRL, ELL, and SPED by Grade and Condition 14 Table 4c. Fastbridge Analysis Sub-Sample: Percent of Students Male, Ethnic Minority, FRL, ELL, and SPED by Condition 14 Table 5. BOY PRE-PSM Reading Overall Scale Scores by Grade and Condition 15 Table 6. BOY POST-PSM Reading Overall Scale Scores by Grade and Condition 16 Table 7a. Star Literacy Benchmark Categories for Beginning of Year 17 Table 7b. Star Reading Benchmark Categories for Beginning of Year 17 Table 7c. FAST ELA Test Level for Beginning of Year 18 Table 8a. Star Early Literacy Skills Assessed and Example Types of Questions 19 Table 8b. Star Reading Domains & Skills 20 Table 8c. FAST Reading Indicators of Early Literacy Skills 20 Figure 1. Number of Weeks Participated (All Participants) 23 Table 9. List of Assessments by Grade Level by Time Period Sample 25 Table 10. Number of Teachers, Schools, and Logs Recorded by Grade 26 Table 11: Description of RISE Usage from Logs overall and by grade level 26 Table 12: Description of Usage from Logs by School 27 Figure 2: Comparing number of stations completed to total unique skills covered by grade 28 Table 13a: Number of Students by Grade with FastBridge WRCPM Scores 29

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Table 13b: Overall impact of RISE on FastBridge WRCPM Scores 30 Figure 3: RISE vs. Comparison Group growth in FastBridge Words Read Correctly Per Minute (WRCPM) Scores 30 Table 14. T-tests comparing Grade Level Growth in Literacy Scores by Treatment and Comparison Group Status 32 Table 15. T-tests comparing Grade Level EOY Literacy Scores by Group Status 33 Figures 4a-b. Scatterplots and trend lines with correlations between students’ fnal lesson number and EOY FAST score or percentile 34 Figure 5a. Student Advancement in Benchmark or Achievement Levels from BOY to EOY 35 Chart 35 Table 14a: Overall Table That shows changes in performance levels on Star Grade 1 (N=34) 35 Figure 6a. First Grade Graph with BOY and EOY Benchmark Categories 36 Table 14b: Changes in performance Category on Star, Grade 2 (N=33) 36 Figure 6b. Second Grade Graph with BOY and EOY Benchmark Categories 37 Table 14c: Overall Table That shows changes in performance levels on FAST (N=101) 37 Figure 6c. Grades 3-5 Graph with BOY and EOY Performance Levels 38 Figure Set 7: Graphs from the RISE Survey are included below 39

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Introduction Strong literacy skills developed in pre-k, kindergarten, and frst grade build a foundation Upon which future academic success thrives. The ability to read, write, and interpret text are integral skills students utilize to advance through their academic careers. Specifcally, reading fuency is a skill that allows students to free up their cognitive load to focus more on constructing meaning than decoding words (Rasinski, 2017). Instructional continuity disruptions throughout the COVID-19 pandemic contributed to an average learning gap of 2.5 months in ELA skills, according to benchmark results from Fall 2019 to Winter 2021 (Education Analytics, 2021). Addressing unfnished learning gaps in various ELA skills as early and efciently as possible is a major concern of those invested in education. Due to learning interruptions throughout the pandemic, the need for efective supplemental reading resources has hugely increased as schools attempt to regain a sense of post-pandemic academic normalcy. Changes in the delivery and rate of learning sessions throughout virtual learning led to a demonstrable impact on overall student reading achievement. Research shows that instruction must go

beyond a single school year of standards to address the varying needs of students in the classroom while pulling Up students who have fallen behind (Lambert & Sassone, 2020). Repeated reading interventions are highly efective in promoting reading fuency as this technique can increase word accuracy, word recognition, and reading speed (Aldhanhani & Abu-Ayyash, 2020; Stevens, Walker, & Vaughn, 2017). Additionally, repeated reading interventions have increased reading comprehension (Cotter, 2012).

Overview of the Program The Scholastic RISE and RISE Up program is a Tier 2 intervention program designed for students in grades 1-5 that provides targeted small-group instruction to address and prevent reading gaps using explicit, structured comprehension, word study, phonics instruction, and guided writing with a recommended usage of 45-60 minutes per day, with one to four teachers. Scholastic RISE is leveled C-N and targets comprehension, word-solving, fuency, and writing. In RISE, groups of four students rotate through four instruction stations for 45-60 minutes per day for six to eight weeks. RISE Up is leveled O-Z and is focused on advancing student comprehension. In RISE Up, groups of

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

four students rotate through three instructional stations focused on comprehension strategies on a short text for 45 minutes a day for six to eight weeks. Scholastic RISE and RISE Up ofer a kit with teacher resources, books, short reads, and digital access for teachers and students. Scholastic partnered with LXD Research to conduct a third-party evaluation of the Scholastic RISE and RISE Up Program as it was implemented in a Florida school district during the 2022-2023 school year. All the elementary schools use Benchmark Advance as a core reading curriculum, and the comparison schools use School Specialty’s SPIRE or Fountas and Pinnell’s Leveled Literacy Intervention. RISE coaches provided guidance for the district to modify the program for use during a 30-minute a day timeframe, with one instructor, to meet the district’s intervention model. Treatment Group: Program Key Features The Scholastic RISE and RISE Up feature instructional practices that difer from the typical reading instruction provided by the core curriculum. A phonemic awareness and phonics continuum of skills is followed using structured literacy characteristics. An emphasis on peer discussions and independent writing during enrichment lessons encourages student voice and choice throughout courses. Texts feature relatable, diverse characters that deal with real-life situations, a strategy enabling students to synthesize previous knowledge with that learned during their lessons to efectively participate in small group discussions where their unfnished learning may have rendered them silent in similar whole group settings.

Table 1. Structured Literacy Characteristics in RISE and RISE Up Lessons

Structured Literacy Characteristic

Associated Scholastic RISE and RISE Up Resource or Instructional Practice

Phonology

Oral segmentation, sound identifcation practice

Syllables

Syllable counting, pattern recognition, open and closed syllables

Morphology

Word families, afxes, base words, morphemes, letter scrambles

Syntax

Sentence building, dictation

Semantics

Context clue exercises, vocabulary, comprehension cards

Sound-Symbol Association

Sound walls, hand signals for letter sounds & blends

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Core Reading Program The district uses the Florida Benchmark Advance core ELA curriculum, published by Benchmark Education. EdReports evaluated Benchmark Advance in 2021 and found Benchmark Advance materials partially met the expectations of alignment (Ed Report, 2021). The program is designed for grades K-6 and is described as a highly fexible reading program with components available in print and online. It features authentic literature, informational texts, and titles from the Florida Book List and focuses on foundations, reading, vocabulary, and communication more than phonics or writing.

Research Questions The evaluation aims to answer the following questions:

1) How do changes in academic outcomes (performance on the Star, FAST, and FastBridge assessments) achieved by RISE/Up students compare to those achieved in the business-as-usual supplemental (Tier 2) intervention course from the middle to the end of the year? 2) How do RISE/Up student performance levels change from the middle to the end of the year?

3) How did interventionists and teachers implement the program? 4) What are teachers’ and students’ perceptions of RISE/Up? 5) What recommendations do teachers have to improve RISE/Up?

Research Design This study used a mixed-methods approach, including a matched quasi-experimental design complemented by classroom observations, teacher surveys, and interviews with school leaders and interventionists. This combination of methods allowed researchers to understand how the materials are being used in the classroom, learn teacher feedback, and the perceived impact of the program while also understanding academic achievement. Scholastic RISE and RISE Up were being implemented in an ethnically diverse school district in Florida. The district serves a population in which 36% were Hispanic/Latino and 28% were Black or African American. Nearly half (47%) of students qualify for free lunch. There were over 73,900 students in grades 1-5 across over 100 elementary schools.

Twenty schools volunteered for the Scholastic RISE and RISE Up Program with Tier 2 students. Coaches collaborated with interventionists to determine which students would be most

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

appropriate to use RISE or RISE Up for Tier 2 1 , due to their own individual needs. In February 2023, interventions for every treatment school provided the district with the student identifcation numbers for all students reviewing RISE or RISE Up during 2022-2023. As expected, the assigned students were a relatively small proportion of Tier 2 students assigned to the program. The district leaders allowed researchers to identify comparison schools that would most closely match the treatment (volunteer) schools using school size, ELA scores from previous years, and demographic profles. There were seven comparison schools included in the study, and since none of these schools would be able to use RISE or RISE Up many more students were available for the comparison group. In exchange for their participation, district leaders received a combination of discounted materials and training. Having a larger comparison group allowed for better matching of students across the district sample in order to establish baseline equivalence and increase the rigor of this study. Sample Because of the successful recruitment of treatment schools, 18-60 students used RISE in each grade (1-5). Second and third grade had the most students, which best aligns with the recommended placement for RISE. Fewer students in fourth and ffth grade were assigned to RISE or RISE Up.

Table 2a. Number of Interventionists are in the Treatment and Comparison Groups Group Number of Instructors Number of Students

Treatment

143

Comparison

109

630

Total

154

773

1 The study used the district’s intervention process outlined in their Reading Intervention handbook, which includes Decision Trees and 17-page Decision Tree Guides for each grade level including FAST and STAR performance requirements. Students not already on an IEP, ELL Plan could participate.

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Table 2b. Number of Students and Schools per Grade and Condition

Number of Students

Number of Schools

Grade

Condition

Treatment

1stGrade

Comparison

Treatment

2ndGrade

Comparison

101

Treatment

3rdGrade

Comparison

Treatment

4thGrade

Comparison

Treatment

5thGrade

Comparison

Treatment

168

Overall

Comparison

330

Matching Procedures

To ensure baseline equivalence, LXD Research applied Propensity Score Matching (PSM) procedures to construct a matched sample of students from the full comparison group, using PSM procedures in the PSM plug-in for SPSS Version 28.0 (Bertsekas & Tseng, 1988; Hansen, 2004: Ho, Imai, King, & Stuart, 2011; Thoemmes & Liao, 2013). PSM is based on logistic regression, with the outcome specifed as a dichotomous indicator of whether or not a student was in the treatment condition. PSM procedures were conducted separately for grades 1 and 2 because the tests for these grades have diferent subtests at the start of the year for each grade level. PSM was conducted for grades 3-5 together because all students took FAST, which has grade-agnostic scale scores 2 . The propensity 2 Combining the grades created a larger overall treatment group to support evaluating 3-5 in this study. Like FAST, RISE instructional levels are based on student skill level and not grade level. In other words, the assessment scores students based on their abilities using a unifed score across grades. Analysis confrmed that each student in a grade had at least one other student in their grade in the sample. Additionally, each grade level BOY scores were statistically similar.

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

score for each student in the dataset corresponded with the likelihood of treatment assignment, given a vector of data elements likely related to outcome or treatment participation. Baseline scores from Renaissance Star Early Learning, Star Reading, and Cambium F.A.S.T. (FAST), and all available student-level demographic data elements were included in the propensity score matching procedure. In addition to the grade-level-appropriate literacy baseline scores, student-level covariates included: gender; racial/ethnic minority; English language learner (ELL) status; special education status; Free/Reduced Lunch (FRL) status. Next, the PSM matching algorithm was applied to select the matched comparison group of students from the original comparison group. LXD Research created the fnal matched sample by considering one-to-one, nearest neighbor matching with a caliper and without replacement. Propensity scores and covariates were evaluated to balance the treatment and comparison groups. Analysts conducted robustness checks by using variations on original propensity score parameters to ensure the most appropriate propensity score matching algorithm was used, as defned by the most balanced observable characteristics between treatment and comparison students. Due to the relatively small sample size of the RISE treatment group, only comparison group students were eliminated from the sample for matching. The results of this matching can be seen below in Tables 3-4. Demographics Groups Before and After Matching and Baseline Equivalence The analytic samples created for this study using the propensity score matching process demonstrate sufcient baseline equivalence within the acceptable range for the What Works Clearinghouse and the Evidence for ESSA website (with efect size statistics lower than .25 of a standard deviation). Characteristics of the student sample before and after matching are presented below. Additional tables presented in the Results section include grade-specifc results for each measure. Nine participants were removed before the PSM due to missing scale score data. Zero treatment and 78 comparison group participants were removed from the fnal data set due to Propensity Score Matching. For details on the grade-level sample sizes, see Table 1.

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Table 3. Number of Students per Grade and Condition Before and After PSM

Grade

Condition

Number of Students Pre-PSM Number of Students Post-PSM

Treatment

1stGrade

Comparison

Treatment

2ndGrade

Comparison

101

Treatment

3rdGrade

Comparison

Treatment

4thGrade

Comparison

Treatment

5thGrade

Comparison

Treatment

168

Overall

Comparison

330

252

Table 4a. PRE-PSM percent of Students Male, Ethnic Minority, FRL, ELL, and SPED by Grade and Condition

Free/ Reduced Lunch

English Language Learner

Number of Students

Gender (Male)

Ethnic Minority

Special Education

Grade

Condition

71%

44%

Treatment

56%

32%

12%

1stGrade

83%

74%

Comparison

57%

28%

13%

Treatment

58%

48%

30%

67%

15%

2ndGrade

Comparison

46%

59%

21%

64%

16%

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

78%

33%

35%

Treatment

101

63%

67%

3rd-5th Grade

69%

21%

24%

Comparison

176

59%

67%

Shaded boxes indicate signifcant or trending diferences between groups

Table 4b. POST-PSM percent of Students Male, Ethnic Minority, FRL, ELL, and SPED by Grade and Condition

Free/ Reduced Lunch

English Language Learner

Number of Students

Gender (Male)

Ethnic Minority

Special Education

Grade

Condition

Treatment

56%

71%

44%

32%

12%

1stGrade

Comparison

57%

75%

56%

31%

13%

Treatment

58%

67%

48%

30%

15%

2ndGrade

Comparison

52%

64%

53%

26%

15%

Treatment

101

63%

78%

67%

33%

35%

3rd-5th Grade

Comparison

139

55%

81%

73%

27%

31%

Table 4c. Fastbridge Analysis Sub-Sample: Percent of Students Male, Ethnic Minority, FRL, ELL, and SPED by Condition

Free/ Reduced Lunch

English Language Learner

Number of Students

Gender (Male)

Ethnic Minority

Special Education

Grade

Condition

Treatment

60%

91%

82%

31%

Grades 1-5

Comparison

213

59%

80%

68%

43%

16%

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Group Literacy Scores Before and After PSM for Star and FAST

Star and FAST Beginning-of-Year The matching students successfully created similar treatment and comparison groups in each grade. The signifcance level for each pair was greater than 0.05, meaning the groups were similar. Table 5 shows the group scores before the matching and Table 6 shows the group scores after the matching.

Table 5. BOY PRE-PSM Reading Overall Scale Scores by Grade and Condition

Grade

Condition

Number of students

Reading Test Score

Treatment

694.1

1stGrade STAR Literacy Unifed Scale Score

Comparison

686.0

Treatment

755.3

2ndGrade STAR Reading Unifed Scale Score

Comparison

753.0

Treatment

268.8

3rdGrade FAST ELA Scale Score

Comparison

266.8

Treatment

278.0

4thGrade FAST ELA Scale Score

Comparison

282.2

Treatment

285.5

5thGrade FAST ELA Scale Score

Comparison

285.0

Treatment

101

273.9

3rd-5th Grade FAST ELA Scale Score

Comparison

176

274.3

Shaded boxes indicate signifcant diferences between groups.

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Table 6. BOY POST-PSM Reading Overall Scale Scores by Grade and Condition

Reading Test Score

Grade

Condition

Number of students

Signifcance

694.1

Treatment

1st grade STAR Literacy Unifed Scale Score 2ndgrade STAR Reading Unifed Scale Score

p =0.96

693.6

Comparison

755.3

Treatment

p =0.97

756.0

Comparison

268.8

Treatment

3rdgrade FAST ELA Scale Score

p =0.70

267.8

Comparison

278.0

Treatment

4thgrade FAST ELA Scale Score

p =0.28

282.3

Comparison

285.5

Treatment

5thgrade FAST ELA Scale Score

p =0.63

283.2

Comparison

101

273.9

Treatment

3rd-5th grade FAST ELA Scale Score

p =0.94

139

273.7

Comparison

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Benchmark Status for Matched Groups Additional descriptive statistics that can be helpful to understand student beginning-of-year (BOY) scores are the Benchmark Status distributions for each grade (Figure 7a-b). As expected, most students selected for intervention scored in the Urgent Intervention or Intervention groups on Star Literacy. Over 80% of third-ffth graders were Below Benchmark on FAST.

Table 7a. Star Literacy Benchmark Categories for Beginning of Year

Number of Students

Urgent Intervention

At/Above Benchmark

Grade

Condition

Intervention On Watch

Treatment

38%

32%

24%

1st grade

Comparison

31%

50%

12%

Table 7b. Star Reading Benchmark Categories for Beginning of Year

Number of Students

Urgent Intervention

At/Above Benchmark

Grade

Condition

Intervention On Watch

Treatment

64%

21%

2ndgrade

Comparison

63%

30%

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Table 7c. FAST ELA Test Level for Beginning of Year

Number of Students

Below Benchmark

At/Near Benchmark

Above Benchmark

Grade

Condition

Treatment

85%

13%

3rdGrade

Comparison

90%

10%

Treatment

83%

17%

4thGrade

Comparison

89%

11%

Treatment

89%

11%

5thGrade

Comparison

93%

Treatment

101

85%

14%

Grades 3-5 Combined

Comparison

139

91%

Site Visit Sample LXD Research began reaching out to schools in early spring 2023 to conduct site visits. Both treatment and comparison schools were largely located near each other in the urban center. One treatment school approximately 40 miles away from the others, an area considerably more rural than the other campuses included in our site visits. Researchers visited fve schools to view RISE/Up implementation. Researchers visited two schools not implementing RISE/Up. Researchers observed an average of 4 students per school ranging from grades 2nd-5th.

Data and Measures

Description of Achievement Data All students were pretested within the frst four weeks of school using Renaissance Star Early Learning (Grades K-2) and Cambium F.A.S.T. (Grades 3-10). Mid-year testing took place in December 2022, and end of year testing followed in April/May 2023. A subgroup of eight treatment schools and all of the comparison schools also completed teacher surveys and used FastBridge to monitor student progress between January and May 2023 (the treatment school teachers also completed weekly logs during this time to track their implementation for each group). A portion of these schools were also visited by the research team in March 2023 for classroom observations.

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Progress Monitoring & Assessment: Star Early Literacy Star Early Literacy Assessment is a computer-based adaptive assessment to measure Pre-K - 3rd early language and reading foundational skills. All questions are in multiple choice format with three answer options. It uncovers learning gaps quickly to identify at-risk students and assess individual growth. Students automatically progress to Star Reading once they reach a certain threshold of mastery. Most of the frst graders in this study met that threshold and advanced to Star Reading for the end-of-year assessment. For more information about the types of skills assessed refer to Table 8a.

Table 8a. Star Early Literacy Skills Assessed and Example Types of Questions

Skill

Examples of Skills Assessed

Distinguish numbers from letters Identify number of words Match words that are the same Identify rhyming words Blend 2-syllable words Recognize same fnal sounds (pictures) Match short vowel sounds to letters Identify letter for fnal consonant sound Identify sounds with word families Read grade-level sight words Understand position words Match words with their synonyms

Alphabetic Principle, Concept of Word, Visual Discrimination

Phonemic Awareness

Phonics

Structural Analysis & Vocabulary

Sentence & Paragraph Level Comprehension

Choosing the right word for the sentence Answering multiple choice comprehension question about text

Progress Monitoring & Assessment: Star Reading Star Reading is for K-12 students to assess their reading literacy growth. The test uses multiple choice short comprehension and extended comprehension question types to assess skills across fve diferent domains as shown in Table 8b.

Table 8b. Star Reading Domains & Skills

Domain

Specifc Skills

Vocabulary

Word knowledge and skills - knowledge and ability to apply

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Comprehension strategies and constructing meaning - self-monitoring, making predictions, drawing conclusions, and using the organizational structure of the text to improve understanding Analyzing literary text - explore the plot, setting, character, theme, point of view, characteristics of diferent genres Understanding author’s craft - understanding and analysis of an author’s use of language and literary devices to create certain efects.

Comprehension

Literacy

Author

Analyzing Argument and Evaluating Text - recognizing, analyzing, and evaluating arguments in persuasive, editorial, and academic writing

Argument

Progress Monitoring & Assessment: FAST Reading The Florida Assessment of Student Thinking (FAST) is a progress monitoring tool for students in Pre-Kindergarten through grade 10 for the reading assessment and through grade 8 for mathematics assessment. Refer to Table 8c for the skills tested on the FAST reading assessment.

Table 8c. FAST Reading Indicators of Early Literacy Skills

Subtest

Indicators of Early Literacy Skills

Reading Prose & Poetry, Reading Informational Text, Reading Across Genres & Vocabulary

FAST ELA Reading

Progress Monitoring & Assessment: FastBridge FastBridge is described as an all-in-one screening, progress monitoring, and reporting tool. Noting that their screening process takes anywhere from fve to thirty minutes for educators to complete with students, FastBridge provides subject mastery data and intervention guidance to educators covering reading, math, and social-emotional behavior. This program provides tier one program evaluation through universal screening, allowing teachers to adeptly identify efective and alter inefective instructional practices. Description of Log Data An additional task asked of the 14 participating RISE/Up teachers was to complete weekly logs from February 6th, 2023 through May 12, 2023. These logs tracked student attendance; lesson number; and specifc details about daily small group lessons, such as station number, resources used, station card, book title, target skill and any other relevant notes the teachers provided. Teachers received a weekly incentive of a $10 Amazon gift card for each log submission, and double the amount

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

if they completed all of their logs by the end of each month. All of our RISE/Up instructors participated and monitored their weekly intervention groups through logs.

Description of Teacher Surveys A survey was sent out to the 14 RISE/Up teachers who opted to participate in the survey and 36 comparison teachers via email. All RISE/Up teachers and 18 comparison teachers completed the survey. Teachers who completed the survey were given a $25 Amazon gift card as an incentive. Questions ranged from demographics and background education information to more specifc intervention experience and professional development. Description of Principal Interviews LXD Research conducted interviews with administrators from treatment campuses. All of our principals had previous teaching experience, and their knowledge of educator sentiments provided us with insights that informed our teacher logs, site visits, and teacher surveys. Like the teachers we interviewed, principals held an overall positive view of RISE/Up. They felt that the program was a helpful resource to incorporate into their school community, teachers were well-trained and supported, and students remained engaged throughout interventions. Program improvement suggestions from administrators appear later in this report. Description of Teacher Observations LXD Research completed site visits in early March for both RISE/Up and comparison classrooms. Two researchers visited 5 RISE/Up schools where they observed 6 teachers using the program with their students. The two researchers also visited 2 comparison schools where they observed 2 teachers using other Tier 2 interventions such as Fountas and Pinnell’s Leveled Literacy Intervention. The site observations allowed LXD Research to witness the program in action, see how the students and teachers were using and responding to it, as well as hear from teachers directly about the positives and challenges of implementing the program. A summary of the observations were shared with the district and Scholastic in a separate report.

Analytical Approach

Time Period of Analysis Interviews with district leadership, conversations with educators during site visits, and analysis of teacher surveys showed that student learning during the fall semester was disrupted multiple times.

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

The research study was approved by the school district in mid-September, which coincided with massive teacher shortages throughout the schools. Shortly after the teachers were trained on RISE, the category 4 hurricane Ian hit the Florida county. A few weeks later in early November, category 1 hurricane Nicole inficted signifcant damage and closed the schools for multiple days. With Thanksgiving break and winter break just weeks later, measuring the impact of RISE between January and May would more accurately refect its efectiveness. This period also coincided with the study’s weekly implementation log collection, providing the researchers with better insight into implementation and instructional patterns to triangulate with quantitative results. Information about the Fall to Winter results were shared with the school district and Scholastic in March 2023 and are available Upon request. Correlation between Lesson Achievement and Literacy Scores Fourteen RISE instructors provided weekly logs documenting their implementation of the program. Each group’s fnal lesson number in May 2023 was recorded and merged with the student literacy score data. A correlation between each student group’s lesson number and their EOY scale score was analyzed, controlling for students BOY scale scores. FastBridge Analytic Approach During the spring semester of 2023, RISE students were asked to participate in a biweekly FastBridge assessment. As expected, most students who took FastBridge had between six to eight time periods (Figure 1). Of the 2779 total responses recorded, 187 responses were redundant (i.e., the same participant responded more than once within the same week). To account for this redundancy, the highest scores for a participant for any given week were included in the fnal data set, and the redundant responses were excluded from analysis. When there is a relatively small sample (45 students in the treatment group) and students have multiple timepoints across the study period, a Repeated Measures ANOVA helps determine whether the trends of scores over time are signifcantly diferent between groups and what kind of pattern or trajectory of growth is shown. Missingness by Participant Of the 378 individual participants (60 RISE students, and 318 comparison students), students varied in their level of participation. To determine the level of within-student missingness, we counted the number of weeks in which each student participated to establish a cutof for students who did not sufciently participate in the longitudinal assessment to be included in the fnal data set. Teachers were requested to evaluate their students at least once every 2 weeks during the 18 week period, with the exception of the spring break week (i.e., Week 10) and state testing (i.e., Weeks 15-18). Students ranged

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

in participation from 1-16 total weeks of measurement, with a mean of 6.9 occasions, a median of 7 occasions, and a mode of 8 occasions overall. The RISE participants had a mean of 6.6 occasions, a median of 7 occasions, and a mode of 6 occasions. The Comparison participants had a mean of 6.9 occasions, a median of 7 occasions, and a mode of 8 occasions (please see Figure 1).

Figure 1. Number of Weeks Participated (All Participants)

Total Weeks Participated

The result of this analysis was to set the participation cutof in the longitudinal data set to 6+ occasions of measurement. 258 of the 378 total participants (i.e., 68%) were therefore included in the longitudinal analysis (45 RISE participants, and 213 Comparison group participants). Participants ranged from Grades 1-5 (see Appendix 1). The diference in attrition between RISE and the comparison group was quite modest (84% to 83% (i.e., -1%) in the comparison group, and 16-17% (i.e., +1%) in the RISE group). Therefore, our approach met the requirements for minimizing diferential attrition in the RISE and Comparison samples. Missingness by Occasion of Measurement There was signifcant missingness in six of the 18 total weeks; Week 1 (73% missing), Week 10 (100% missing), and Weeks 15-18 (74% missing). Week 1 missingness was due to communication issues at launch, Week 10 was spring break, and Weeks 15-18 were interrupted by state testing. Therefore, Weeks 1, 10, and 15-18 were excluded from analysis. In addition, the participants were asked to respond on a biweekly basis, so missingness at the individual level typically followed a biweekly pattern. Therefore, scores were averaged in two-week intervals (e.g., Weeks 2 and 3 were averaged, and Weeks 4 and 5 were averaged). This averaging resulted in a reduction to six occasions of measurement. After

LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report

Page i Page ii Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Page 9 Page 10 Page 11 Page 12 Page 13 Page 14 Page 15 Page 16 Page 17 Page 18 Page 19 Page 20 Page 21 Page 22 Page 23 Page 24 Page 25 Page 26 Page 27 Page 28 Page 29 Page 30 Page 31 Page 32 Page 33 Page 34 Page 35 Page 36 Page 37 Page 38 Page 39 Page 40 Page 41 Page 42 Page 43 Page 44 Page 45 Page 46 Page 47 Page 48 Page 49 Page 50 Page 51 Page 52 Page 53 Page 54 Page 55 Page 56 Page 57 Page 58 Page 59 Page 60 Page 61 Page 62 Page 63 Page 64 Page 65 Page 66 Page 67 Page 68 Page 69 Page 70 Page 71 Page 72 Page 73 Page 74 Page 75 Page 76 Page 77 Page 78 Page 79 Page 80 Page 81 Page 82 Page 83 Page 84 Page 85 Page 86 Page 87 Page 88 Page 89 Page 90 Page 91 Page 92 Page 93 Page 94

Made with FlippingBook - Online catalogs