score for each student in the dataset corresponded with the likelihood of treatment assignment, given a vector of data elements likely related to outcome or treatment participation. Baseline scores from Renaissance Star Early Learning, Star Reading, and Cambium F.A.S.T. (FAST), and all available student-level demographic data elements were included in the propensity score matching procedure. In addition to the grade-level-appropriate literacy baseline scores, student-level covariates included: gender; racial/ethnic minority; English language learner (ELL) status; special education status; Free/Reduced Lunch (FRL) status. Next, the PSM matching algorithm was applied to select the matched comparison group of students from the original comparison group. LXD Research created the fnal matched sample by considering one-to-one, nearest neighbor matching with a caliper and without replacement. Propensity scores and covariates were evaluated to balance the treatment and comparison groups. Analysts conducted robustness checks by using variations on original propensity score parameters to ensure the most appropriate propensity score matching algorithm was used, as defned by the most balanced observable characteristics between treatment and comparison students. Due to the relatively small sample size of the RISE treatment group, only comparison group students were eliminated from the sample for matching. The results of this matching can be seen below in Tables 3-4. Demographics Groups Before and After Matching and Baseline Equivalence The analytic samples created for this study using the propensity score matching process demonstrate sufcient baseline equivalence within the acceptable range for the What Works Clearinghouse and the Evidence for ESSA website (with efect size statistics lower than .25 of a standard deviation). Characteristics of the student sample before and after matching are presented below. Additional tables presented in the Results section include grade-specifc results for each measure. Nine participants were removed before the PSM due to missing scale score data. Zero treatment and 78 comparison group participants were removed from the fnal data set due to Propensity Score Matching. For details on the grade-level sample sizes, see Table 1.
LXD Research -RISE and RISE UP Winter 2022-Spring 2023 Report
12
Made with FlippingBook - Online catalogs