RESEARCH REPORT
An Evaluation of Scholastic Presents LitCamp in Schenectady City School District, NY
August 2019
Prepared by RMC Research Corporation 111 SW Columbia Street, Suite 1030 Portland, OR 97201
ABOUT RMC RESEARCH RMC Research helps state, local, and federal agencies, philanthropic foundations, and creative and social service organizations improve outcomes for their constituents. CONTACT For more information about this report, please contact Scholastic Research & Validation at ScholasticRV@scholastic.com or visit www.scholastic.com/research. SUGGESTED CITATION Beam, M. (2019). An evaluation of Scholastic presents LitCamp in Schenectady City School District, NY. Portland, OR: RMC Research Corporation.
Scholastic is not responsible for the content of third-party websites and does not endorse any site or imply that the information on the site is error-free, correct, accurate, or reliable.
Item #: 691824 TM ® & © Scholastic Inc. All rights reserved.
SCHOLASTIC and associated logos are trademarks and/or registered trademarks of Scholastic Inc.
Other company names, brand names, and product names are the property and/or trademarks of their respective owners. Scholastic does not endorse any product or business entity mentioned herein.
RESEARCH REPORT
An Evaluation of Scholastic Presents LitCamp in Schenectady City School District, NY
August 2019
Contents
Executive Summary...........................................................................................................................1 Evaluation Overview...........................................................................................................................1 Evaluation Data Sources...................................................................................................................2 Student Outcome Findings .............................................................................................................2 Implementation Findings.................................................................................................................. 3 Discussion ........................................................................................................................................... 4
Introduction........................................................................................................................................5
Research Overview............................................................................................................................6 Student Sample................................................................................................................................... 6 Teacher Sample.................................................................................................................................. 6 Family Sample..................................................................................................................................... 7 Student Outcome Findings .............................................................................................................7 Student Reading Knowledge...........................................................................................................7 Student Behaviors and Attitudes.....................................................................................................7 Implementation Findings...............................................................................................................12 Teachers’ Perceptions of LitCamp ...............................................................................................12 Program Implementation ..............................................................................................................14 Student Engagement ......................................................................................................................14
Discussion .........................................................................................................................................15
References......................................................................................................................................... 17
Appendix A LitCamp Activities......................................................................................................18
Appendix B Evaluation Methodology..........................................................................................19
Exhibits
Exhibit 1. Change in Grade 5 Students’ Reading Behaviors ........................................................ 8
Exhibit 2. Effectiveness of LitCamp at Cultivating Students’ Reading Skills ............................. 8
Exhibit 3. Change in Grade 5 Students’ Enjoyment of Reading During Summer Vacation... 9
Exhibit 4. Change in Grade 3 Students’ Courage to Read Challenging Books ..................... 10
Exhibit 5. Effectiveness of LitCamp at Developing Students’ Social-Emotional Skills .......... 11
Executive Summary LitCamp is a summer literacy and enrichment program for students in Grades kindergarten through 8. 1 LitCamp aims to build students’ social-emotional skills around what the program refers to as “the 7 strengths”—belonging, curiosity, friendship, kindness, confidence, courage, and hope—by incorporating these concepts into reading and writing texts and lessons and combining them with an engaging summer camp approach. Scholastic Research & Validation partnered with RMC Research to conduct a third-party evaluation of Scholastic Presents LitCamp as it was implemented 2 in summer 2018 by Schenectady City School District in New York. The district provided two hours of LitCamp instruction daily to students in kindergarten through Grade 6 as the literacy component of its districtwide Summer Enrichment Program.
Evaluation Overview The evaluation aimed to answer the following questions:
1. How did students’ reading knowledge change as a result of their exposure to LitCamp?
2. How did students’ reading behaviors and attitudes toward reading change as a result of their exposure to LitCamp?
3. How did teachers perceive LitCamp?
4. What did LitCamp implementation look like in practice?
Additionally, the evaluation examined the effects LitCamp had on a subsample of students and their families. 3
RMC Research collected and analyzed quantitative data through student (Grades 3–5) and teacher surveys and qualitative data through teacher focus groups, classroom observations, and family interviews. RMC Research also explored reading achievement data shared by the district for students (Grades K–5) who participated in the Summer Enrichment Program who received two hours of LitCamp instruction daily (“LitCamp students”; n = 266) and a statistically similar comparison group of students (“Comparison students”; n = 266) who were not enrolled in the Summer Enrichment Program. Findings presented in this report reflect the analysis of data from students, teachers, and families.
1 Grade corresponds to grade level completed in the spring. 2 LitCamp was designed for flexible implementation—providing 50 hours (20 lessons) of literacy instruction within a 4- to 8-week time frame. 3 See the Case Study titled, "How LitCamp Positively Impacted Students’ Literacy & Social-emotional Learning" at http://scholastic.com/research.
LITCAMP EVALUATION IN SCHENECTADY CITY SCHOOL DISTRICT 1
Evaluation Data Sources
Reading achievement data for students in kindergarten through Grade 5
Student survey respondents in Grades 3–5
Teacher survey respondents who taught kindergarten through Grade 6
Teacher focus group participants who taught kindergarten through Grade 5
Classroom observations in kindergarten through Grade 5
Family interviews with families of students in kindergarten and Grades 1, 4, and 5
Student Outcome Findings
Student Reading Knowledge Analyses of the AIMSweb ®4 Oral Reading Fluency subtest revealed promising trends 5 for LitCamp students warranting further exploration with a larger sample and students spanning the full range of achievement. Subgroup analyses revealed a positive trend for English Learner (EL) students such that EL LitCamp students experienced slightly greater gains on the AIMSweb Oral Reading Fluency subtest from spring to fall than EL students in the comparison group across Grades 1–5. Analyses suggest that LitCamp may be providing students with skills that support their academic performance on standardized tests, but most importantly, additional data gathered show the significant impact LitCamp is having on students’ attitudes and beliefs about themselves as readers. Data discussed below highlight how LitCamp supports students' social- emotional learning through texts in an engaging and encouraging environment. Student Behaviors and Attitudes Information about students’ reading behaviors, attitudes toward reading, and social-emotional learning was collected from student and teacher surveys, teacher focus groups, and family interviews. Students across all grades reported positively about their participation in LitCamp. Findings revealed statistically significant changes between baseline and follow-up for students in Grades 3 and 5:
• Reading behaviors. 94% of Grade 5 students reported “reading different kinds of books” after participating in LitCamp (compared to 81% at the beginning of LitCamp).
4 The AIMSweb is a benchmark assessment that informs instruction to improve achievement. Benchmarks are established three times per year for all students using grade-level assessment probes. Reports identify students at risk, help focus areas of individualized instruction, evaluate student progress, and serve as an accountability and communication tool for system improvement. For more information on the assessment, please visit https://www.AIMSweb.com/. 5 Trends are promising in the positive direction but are not statistically significant at the 95% confidence interval.
2
SCHOLASTIC RESEARCH & VALIDATION
• Attitudes toward reading. 48% of Grade 5 students reported liking or loving to “read during summer vacation” after participating in LitCamp (compared to 31% at the beginning of LitCamp). • Social-emotional learning. 86% of Grade 3 students had the courage “[to read] books that might be hard to read” after participating in LitCamp (compared to 67% at the beginning of LitCamp). Teacher surveys, focus groups, and classroom observations offered more nuanced information on the positive impact LitCamp had on students’ reading behaviors, attitudes toward reading, and social-emotional learning. • Reading behaviors. More than three quarters of teacher survey respondents rated LitCamp as effective at cultivating students’ abilities to read independently (84%), choose books (82%), read different kinds of books (82%), and read aloud (80%). These quantitative data were corroborated by teacher focus group participants who reported that students were reading for longer periods of time and choosing more challenging books after participating in LitCamp. • Attitudes toward reading. During focus groups, teachers discussed shifts in students’ attitudes that they attributed to LitCamp, such as students seeming more excited about reading, students being more willing to read, and some students feeling more comfortable reading aloud. • Social-emotional learning. Approximately three quarters of the teacher survey respondents rated LitCamp as effective at developing students’ curiosity (86%), friendship skills (82%), belonging (80%), kindness (80%), confidence (78%), courage (74%), and hope (74%). Teachers also appreciated the social-emotional aspects of LitCamp and the incorporation of the 7 strengths into the book selections. Some teachers mentioned observing a greater sense of community among students, as demonstrated by displays of kindness and empathy. Family interviews provided additional information on the influences of LitCamp, including learning new vocabulary, reading different kinds of books, being more interested in reading, having more patience, and acting more politely.
Implementation Findings
Teacher Perception of LitCamp and Classroom Observations Teacher surveys offered information about teachers’ perception of LitCamp. Information gathered during teacher focus groups and classroom observations provided further context for the teacher survey findings and highlighted teachers' and students' positive experiences with LitCamp.
LITCAMP EVALUATION IN SCHENECTADY CITY SCHOOL DISTRICT 3
Resources. Teacher survey respondents rated the following LitCamp resources very favorably for their utility: Leader’s Guide (92%), book selections (86%), and lessons (86%). In focus groups, teachers made positive comments about the structure and ease of use of the Leader’s Guide , the selection of books, and the variety of activities and writing assignments. Activities. Overall, teachers rated the Read Aloud, Bring the Text to Life, and Bunk Time LitCamp activities as “the most appealing” and believed that their students would agree (see Appendix A for a description of LitCamp activities). This was corroborated by classroom observations that described students as seeming especially engaged during Read Aloud activities and when working independently, as well as appearing excited and responsive when they had opportunities to share their thinking throughout LitCamp. Professional Learning. Overall, teachers felt that the LitCamp professional learning prepared them to implement the program. Implementation. Teachers modified LitCamp activities or altered the schedule of lessons as needed to fit the diverse needs of their students and adapt to the overall Summer Enrichment Program schedule. Discussion A primary goal of Schenectady City School District’s integration of LitCamp into its Summer Enrichment Program was to sustain students’ literacy skills over the summer. The district’s decision to provide two hours of LitCamp instruction daily to students was associated with promising trends for students’ Oral Reading Fluency subtest scores (AIMSweb), in particular for EL LitCamp students (Grades 1–5), who displayed slightly greater gains than the EL students in the comparison group. Most importantly, LitCamp favorably influenced students’ reading behaviors, their attitudes toward reading, and their social-emotional learning as evidenced by both student self-reports and teacher reports, with the majority of teachers reporting that LitCamp was effective at cultivating students’ reading abilities and social-emotional skills. Additionally, teachers indicated confidence in LitCamp through their positive reviews of the LitCamp approach, program materials, and supports. This evaluation aimed to assess how exposure to LitCamp influenced students’ reading behaviors, attitudes toward reading, social-emotional skills, and reading achievement. Examining data from multiple sources, this report highlights and provides context for how LitCamp positively influenced each of these outcomes.
4
SCHOLASTIC RESEARCH & VALIDATION
Introduction LitCamp is a summer literacy and enrichment program for students in Grades kindergarten–8. 6 LitCamp combines reading and writing texts and lessons with an engaging summer camp approach. Through reading, writing, listening, and speaking, LitCamp aims to strengthen students’ literacy skills and affinity for books. Additionally, LitCamp builds students’ social-emotional skills around what the program refers to as “the 7 strengths”—belonging, curiosity, friendship, kindness, confidence, courage, and hope—by incorporating these concepts into LitCamp texts and lessons.
Scholastic Research & Validation partnered with RMC Research to conduct a third-party evaluation of Scholastic Presents LitCamp to answer the following research questions:
1. How did students’ reading knowledge change as a result of their exposure to LitCamp?
2. How did students’ reading behaviors and attitudes toward reading change as a result of their exposure to LitCamp?
3. How did teachers perceive LitCamp?
4. What did LitCamp implementation look like in practice?
Additionally, the evaluation examined the effects LitCamp had on a subsample of students and their families. 7
To address these questions, the evaluation team collaborated with Schenectady City School District in New York, which enrolls nearly 10,000 students in 11 elementary schools, three middle schools, and one high school. 8 The district website reports that the student population is culturally diverse with more than 30 languages represented and that 79% of students come from economically disadvantaged homes. The district first implemented LitCamp in 2017 as part of its Summer Enrichment Program that historically has served up to 1,100 students in Grades kindergarten–6. This 4-week program operates Monday through Friday for 10 hours per day; LitCamp is embedded into each day for approximately two hours, serving as the primary literacy instruction component.
6 Grade corresponds to grade level completed in the spring. 7 See the Case Study titled, "How LitCamp Positively Impacted Students’ Literacy & Social-emotional Learning” at http://scholastic.com/research. 8 Retrieved from: http://www.schenectady.k12.ny.us/about_us/enrollment (October 2018)
LITCAMP EVALUATION IN SCHENECTADY CITY SCHOOL DISTRICT 5
Research Overview RMC Research conducted the evaluation in summer 2018, the district’s second summer implementing LitCamp, using a mixed-methods design that involved collecting quantitative data through student and teacher surveys; qualitative data through focus groups, classroom observations, and family interviews; and student reading achievement data from the district. Student and teacher surveys were administered during the first and last weeks of the Summer Enrichment Program, generating baseline (at the beginning of LitCamp) and follow-up (after participating in LitCamp) data that were used to assess changes in attitudes over time and capture students’ and teachers’ perceptions of the program. RMC Research visited all four Summer Enrichment Program sites to observe how LitCamp was being implemented in classrooms and conduct teacher focus groups and family interviews to obtain in-depth information about their respective experiences with LitCamp. A detailed description of the evaluation methodology appears in Appendix B. Student Sample In summer 2018, a total of 971 students were enrolled in the Summer Enrichment Program. RMC Research administered student surveys to a subset of LitCamp students—students in Grades 3–5 only. The survey sample is comprised of 205 students from this subset who completed both a baseline survey (at the beginning of LitCamp) and a follow-up survey (after participating in LitCamp). To further explore student outcomes, RMC Research requested demographic and spring and fall assessment data for all students (Grades K–6 9 ) who participated in the districtwide Summer Enrichment Program (“LitCamp students”; n = 266) and a statistically similar matched comparison group (“Comparison students”; n = 266) comprising the assessment sample . 10 Teacher Sample In summer 2018, a total of 80 teachers implemented LitCamp in kindergarten through Grade 6. RMC Research administered a survey to all LitCamp teachers to gather information about how they implemented LitCamp and their impressions of LitCamp. A subsample of 50 teachers completed an end-of-session survey at the end of LitCamp (comprising the survey end-of-session sample ). RMC Research also conducted classroom observations and teacher focus groups at each of the four sites. In total, RMC Research observed 11 classrooms at different grade levels and spoke with 21 teachers spanning kindergarten through Grade 5 during focus groups. 11
9 Grade corresponds to grade level completed in spring 2018. 10 The district changed assessments between spring and fall 2018, and the district only administered AIMSweb ® to approximately one-third of all students in fall 2018. Tested students were those who scored in the lowest tier in spring 2018 and are represented in the 266 students who make up the LitCamp student assessment sample and 266 students who make up the Comparison student assessment sample. 11 The study did not target Grade 6 because few Grade 6 students participated in the Summer Enrichment Program and only at one site.
6
SCHOLASTIC RESEARCH & VALIDATION
Family Sample RMC Research conducted semi-structured interviews with four families of children that attended the Summer Enrichment Program.
Student Outcome Findings Student Reading Knowledge
RMC Research used reading achievement data from district-administered student assessments to explore LitCamp students’ change in reading knowledge compared to a statistically similar matched group of students that did not participate in the Summer Enrichment Program.
Oral Reading Fluency Analyses of the AIMSweb ®12 Oral Reading Fluency subtest revealed promising trends 13 for LitCamp students warranting further exploration with a larger sample and students spanning the full range of achievement. Subgroup analyses revealed a positive trend for English Learner (EL) students such that EL LitCamp students experienced slightly greater gains on the AIMSweb Oral Reading Fluency subtest from spring to fall than EL students in the comparison group across Grades 1–5. Analyses suggest that LitCamp may be providing students with skills that support their academic performance on standardized tests, but most importantly, additional data gathered show the significant impact LitCamp is having on students’ attitudes and beliefs about themselves as readers. Data discussed below highlight how LitCamp supports students' social-emotional learning through texts in an engaging and encouraging environment. Student Behaviors and Attitudes RMC Research used student survey data to assess changes in students’ behaviors and attitudes toward reading and students’ social-emotional learning. Teacher surveys and focus groups offered information on teachers’ perceptions about students’ reading behaviors, attitudes toward reading, and social-emotional learning. Additionally, family interviews provided information on the influence of LitCamp on students’ reading behaviors at home. Students across all grades reported positively about their participation in LitCamp.
12 AIMSweb is a benchmark assessment that informs instruction to improve achievement. Benchmarks are established three times per year for all students, using grade-level assessment probes. Reports identify students at risk, help focus areas of individualized instruction, evaluate student progress, and serve as an accountability and communication tool for system improvement. For more information on the assessment, please visit https://www.AIMSweb.com/. 13 Trends are promising in the positive direction but are not statistically significant at the 95% confidence interval.
LITCAMP EVALUATION IN SCHENECTADY CITY SCHOOL DISTRICT 7
Reading Behaviors
STUDENTS—Survey | Students described their respective reading behaviors using a rating scale from 1 ( not at all like me ) to 4 ( a lot like me ). After participating in LitCamp, 94% of Grade 5 students reported “reading different kinds of books” —a statistically significant increase from 81% at the beginning of LitCamp; see Exhibit 1.
Exhibit 1. Change in Grade 5 Students' Reading Behaviors After participating in LitCamp, significantly more Grade 5 students rated “reading different kinds of books” a little like me or a lot like me than at the beginning of LitCamp.
Baseline
81%
Follow-up
94%
n = 52
TEACHERS—Survey | Teachers rated how effectively LitCamp cultivated students’ reading behaviors on a scale from 1 ( not at all effective ) to 5 ( extremely effective ). Exhibit 2 shows that of the 50 teachers who completed an end-of-session survey, over three quarters rated LitCamp as effective at cultivating students’ reading independently (84%), choosing books (82%), reading different kinds of books (82%), and reading aloud (80%)—with a high percentage of teachers rating LitCamp as very or extremely effective at cultivating students’ ability to read aloud (52%), read independently (48%), read different kinds of books (48%), and choose books (42%).
Exhibit 2. Effectiveness of LitCamp at Cultivating Students' Reading Skills Most teachers rated LitCamp as effective at cultivating students' reading abilities:
84%
82%
82%
80%
Read independently
Choose books
Read different kinds of books
Read aloud
8
SCHOLASTIC RESEARCH & VALIDATION
TEACHERS—Focus Groups | Teachers who participated in the focus groups reported observing students develop fluency and read books for longer periods of time , and some teachers commented that students were choosing more challenging books . One teacher specifically noted that students were using vocabulary introduced in the LitCamp lessons in discussions. Another teacher reported increased confidence in reading among some students.
"We're working on vocabulary and—if we talk about a vocabulary word—a couple of my [ students ] will use that word throughout or make a connection.”
– GRADE 4 TEACHER
FAMILIES—Interview | During a family interview, a Grade 5 student reported reading different kinds of books such as graphic novels, and being exposed to new vocabulary because of LitCamp.
Attitudes Toward Reading
STUDENTS—Survey | Students rated how they felt about a series of reading-related activities using a scale from 1 ( I don’t like it! ) to 4 ( I love it! ). After participating in LitCamp, 48% of Grade 5 students reported liking or loving to read during summer vacation —a statistically significant increase from 31% at the beginning of LitCamp; see Exhibit 3.
Exhibit 3. Change in Grade 5 Students' Enjoyment of Reading During Summer Vacation After participating in LitCamp, significantly more Grade 5 students said I like it or I love it! to reading during summer vacation than at the beginning of LitCamp.
Baseline
31%
Follow-up
48%
n = 52
TEACHERS—Focus Groups | Teachers noticed that older grade 14 students were more willing to read . One teacher reported that after participating in LitCamp, some students seemed more excited about reading , and another teacher reported students became more comfortable reading aloud .
"They are more comfortable . . . whereas at the beginning our kids wouldn’t read out loud to me at all, now I’ll start reading it and get them into it and [ then they are saying ] , “Can I read? Can I read?”"
– GRADE 5 TEACHER
14 Generally, older grade refers to Grade 3 through 6.
LITCAMP EVALUATION IN SCHENECTADY CITY SCHOOL DISTRICT 9
FAMILIES—Interview | During a family interview, one set of parents stated that they noticed changes in their younger child’s interest in reading as evidenced by the child bringing home books to read at bedtime, in contrast to previously not showing much interest in reading.
"My kids are more talkative. When we [ discuss ] the stories, the ones that usually are quiet or shy raise their hand and say, "Oh, I like that," or "That relates to me." They'll talk more in the circle when we're doing the story."
– KINDERGARTEN TEACHER
Social-Emotional Learning—The 7 Strengths
STUDENTS—Survey | The student survey included items related to the 7 strengths incorporated into LitCamp—belonging, curiosity, friendship, kindness, confidence, courage, and hope—and students rated statements related to these strengths on a scale from 1 ( not at all like me ) to 4 ( a lot like me ). After participating in LitCamp, 86% of Grade 3 students reported having the courage “to read books that might be hard to read” — a statistically significant increase from 67% at the beginning of LitCamp; see Exhibit 4. Exhibit 4. Change in Grade 3 Students' Courage to Read Challenging Books After participating in LitCamp, significantly more Grade 3 students rated having the courage “to read books that might be hard to read” as a little like me or a lot like me than at the beginning of LitCamp.
Baseline
67%
Follow-up
86%
n = 72
TEACHERS—Survey | Teachers used a scale from 1 ( not at all effective ) to 5 ( extremely effective ) to rate LitCamp’s effectiveness in developing the 7 strengths in students. Exhibit 5 shows that of the 50 teachers who completed an end-of-session survey, a majority rated LitCamp as effective at developing students’ curiosity (86%), friendship (82%), belonging (80%), kindness (80%), confidence (78%), courage (74%), and hope (74%)—with nearly half of the teachers rating LitCamp as very or extremely effective at developing students’ kindness (52%), friendship skills (48%), and confidence (46%).
10
SCHOLASTIC RESEARCH & VALIDATION
Exhibit 5. Effectiveness of LitCamp at Developing Students' Social-Emotional Skills A majority of teachers rated LitCamp as effective at cultivating students' social-emotional skills:
80%
80%
86%
82%
Curiosity
Friendship
Belonging
Kindness
78%
74%
74%
Confidence
Courage
Hope
"It opens dialogue and the ability for them to continue from what they’ve read in the book, allowing them to speak… They’ll start talking to me about how this [ kid ] wasn’t nice to this [ kid ] or how, ‘My mom did this,’ and it’s amazing how they’re coming out and talking about things. It’s allowed them to speak more about how they feel, which is important."
TEACHERS—Focus Groups | Teachers appreciated the social-emotional aspects of LitCamp and the incorporation of the 7 strengths into the book selections. Some teachers reported a greater sense of community among students— they were more comfortable sharing during discussions. One teacher mentioned witnessing their students display kindness , having observed students more frequently apologize to one another directly and unprompted. Another teacher asserted that students demonstrated more empathy after associating their own experiences with a particular story. Additionally, one teacher reported that some students were willing to take greater risks in discussions. Several teachers agreed that students were thinking about the 7 strengths—for example, identifying kindness in fellow classmates—even if they had not yet fully internalized the concepts.
– GRADE 5 TEACHER
"I think every one [ of the strengths ] we’ve done, they are looking at those strengths and looking to see in their life if that’s something they have. Even if they’re not really internalizing it, they’re identifying it."
– GRADE 2 TEACHER
LITCAMP EVALUATION IN SCHENECTADY CITY SCHOOL DISTRICT 11
FAMILIES—Interview | All of the families interviewed appreciated LitCamp’s emphasis on cultivating students’ personal strengths. For example, they mentioned the positive example set by literary characters who work through dilemmas relevant to youth. Parents also valued the opportunities to discuss bullying and hurt feelings with their children using the characters and stories from LitCamp texts to contextualize typical childhood experiences in meaningful ways. One set of parents observed that their younger daughter appeared to have more patience and act more politely since participating in LitCamp.
"Because if there’s characters inside of a book that come to a dilemma…it would be easier for my 10-year-old to…read it and understand it and then approach me with her questions, instead of us having to just come up with the scenario that probably would most likely not fit into her everyday life." – MOTHER
Implementation Findings Teachers’ survey responses reflected their perceptions of LitCamp resources before and after they implemented LitCamp and how well the training prepared them to provide LitCamp instruction in the classroom. In addition to information gained from classroom observations, information gathered during teacher focus groups provided further context for the teacher survey findings.
Teachers' Perceptions of LitCamp
LitCamp Resources The LitCamp Leader’s Guide lays out each lesson according to the following activities: Opening Campfire, Read Aloud, Bring the Text to Life, Reading Power, Bunk Time, Community Lit, Writing Power, and Closing Campfire. It lists the resources required and provides prompts for each activity (see Appendix A for a description of each activity). The Leader’s Guide also offers ways to support ELs in various activities. Teachers are encouraged to use the guide but may adapt the activities as needed. Each teacher receives a set of read-aloud books tailored to students’ grade levels.
12
SCHOLASTIC RESEARCH & VALIDATION
TEACHERS—Survey | Teachers rated LitCamp resources very favorably for their utility. A majority of the 50 teachers who completed an end-of-session survey rated the following resources as moderately or extremely useful : Leader’s Guide (92%), book selections (86%), and lessons (86%). Teachers made positive comments about:
• the structure and ease of use of the Leader’s Guide
• the selection of books
• the variety of activities and writing assignments
The survey asked teachers to identify the three LitCamp activities they and their students found most appealing. Overall, teachers rated the Read Aloud, Bring the Text to Life, and Bunk Time LitCamp activities as most appealing (76%, 64%, and 50%, respectively) and believed that their students would also agree (84%, 62%, and 62%, respectively). TEACHERS—Focus Groups | Across sites, teachers generally found the Leader’s Guide to be well organized and easy to follow. One teacher described the book introductions and discussion prompts as especially helpful. Teachers also provided positive feedback specifically on six of the eight LitCamp activities: Opening Campfire Teachers indicated that LitCamp songs and the Word of the Day worked especially well in the younger grades. 15 Read Aloud The majority of teachers in younger grades reported that the Read Aloud activities worked well for their students. Bring the Text to Life Many teachers of older grades described the Bring the Text to Life activities as beneficial and engaging. Reading Power Many teachers of younger grades and some teachers of older grades reported that the graphic organizers were a helpful tool.
Bunk Time Teachers of older students in particular regarded Bunk Time positively.
Writing Power Teachers appreciated that this activity afforded students much-needed writing practice and encouraged students to relate their writing to the texts they had read.
15 Generally, younger grades refers to kindergarten through Grade 2 and older grades to Grades 3 through 6.
LITCAMP EVALUATION IN SCHENECTADY CITY SCHOOL DISTRICT 13
Program Implementation Through teacher surveys and classroom observations, RMC Research gathered information about how LitCamp was implemented. Teachers modified LitCamp activities or altered the schedule of lessons as needed to fit the diverse needs of their students and adapt to the overall Summer Enrichment Program schedule. TEACHERS—Survey | When asked how they adapted LitCamp to fit their schedules and the needs of their students, some teachers reported that they modified activities, for example, by conducting activities with the whole classroom that were originally designated for independent work or extending activities by developing projects complementary to LitCamp’s texts. Examples of altered schedule or pacing included breaking the daily lessons into two parts or changing the order of activities to better suit students’ energy levels over the course of a full day. Student Engagement TEACHERS—Survey | The end-of-session survey asked teachers to rate their perceptions of student engagement on a rating scale from 1 ( not engaged ) to 5 ( extremely engaged ). Overall, teachers perceived students to be most engaged in Read Aloud, Bring the Text to Life, and Bunk Time, though the ratings varied by grade level. For example, kindergarten teachers rated students’ engagement with Writing Power highest, whereas Grade 1 teachers rated student engagement with Reading Power highest. Teachers of kindergarten and Grades 2, 4, and 6 also rated students as highly engaged in Community Lit. CLASSROOMS—Observation | RMC Research observed that students seemed especially engaged during Read Aloud and when working independently and appeared excited and responsive when given opportunities to share their thinking throughout LitCamp. Kindergarten students were the most engaged and attentive grade observed regardless of the activity. In observations in Grade 2, 3, and 4 classrooms, students tended to be more focused during teacher-led activities than independent activities, and Grade 5 students’ engagement varied widely across activities.
14
SCHOLASTIC RESEARCH & VALIDATION
Discussion A primary goal of Schenectady City School District’s integration of LitCamp into its Summer Enrichment Program was to sustain students’ literacy skills over the summer—a time when students, especially those from disadvantaged backgrounds, have a tendency to lose some of the skills and reading achievement gains made during the previous school year (Alexander, Entwisle, & Olson, 2001). The district’s decision to provide two hours of LitCamp instruction daily to students was associated with promising trends for students’ Oral Reading Fluency subtest scores (AIMSweb), in particular for EL LitCamp students (Grades 1–5) who displayed slightly greater gains than the EL comparison students. The LitCamp model used as part of the district’s Summer Enrichment Program generated important findings for reading behaviors and attitudes as well as social-emotional learning. Specifically, after participating in LitCamp, Grade 3 students reported a significant increase in their courage to read books that might be hard to read. Increasing students’ belief in themselves as avid, varied, and confident readers and writers is critical to their later academic success (Graham, Berninger, & Fan, 2007; Liew, McTigue, Barrois, & Hughes, 2008; Unrau & Schlackman, 2006). Furthermore, the skills LitCamp provides can support students’ academic endeavors throughout the school year as well as throughout the summer when resources may be less accessible. Additionally, reading behavior and attitude outcomes were also evident for students in Grade 5. Specifically, the percentage of Grade 5 students reporting reading different kinds of books and liking or loving to read over the summer increased significantly after participating in LitCamp . The increased receptiveness to reading exhibited by these students came the summer before Grade 6—a pivotal time when confidence is critical for helping students through the challenges of adjusting to middle school. LitCamp also succeeded in improving students’ social-emotional skills; a majority of teachers reported that LitCamp was effective at developing students’ curiosity, friendship skills, and kindness along with other social-emotional skills. Additionally, teachers expressed that LitCamp—which seeks to create a respectful community of readers, writers, and listeners— seemed to foster a greater sense of community among students as demonstrated by student displays of kindness and empathy.
LITCAMP EVALUATION IN SCHENECTADY CITY SCHOOL DISTRICT 15
Teachers’ experiences with LitCamp were positive. More than three quarters of teachers believed LitCamp to be effective at cultivating students’ reading abilities . Teachers reported that their students were reading for longer periods of time and choosing more challenging books after participating in LitCamp . The successes experienced as a result of their LitCamp instruction may have provided teachers with the encouragement and confidence to incorporate best practices into their classroom literacy instruction, including development of social-emotional skills and deepening of students’ understanding and ability to relate to story characters. Witnessing their students become avid, varied, engaged readers and writers, and more active members of their school community can be powerful for teachers, especially after completing a full academic year of instruction. LitCamp offered a way to restore their students’ engagement and enjoyment of texts away from the rapid pace of academic-year instruction. In addition to information provided by students and teachers, a subset of families discussed the impact of LitCamp on their children. Parents reported positive impacts on their children ranging from increased interest in reading to increased politeness . Students interviewed with their families also articulated their LitCamp successes, including an interest in reading different kinds of books and being exposed to new vocabulary . Families found considerable value in LitCamp’s focus on the 7 strengths and capitalized on the social-emotional themes in the various texts to discuss difficult topics with their children. This evaluation aimed to assess how exposure to LitCamp influenced students’ reading behaviors, attitudes toward reading, social-emotional skills, and reading achievement. Examining data from multiple sources, this report highlights and provides context for how LitCamp positively influenced each of these outcomes.
16
SCHOLASTIC RESEARCH & VALIDATION
References
Alexander, K. L., Entwisle, D. R., & Olson, L. S. (2001). Schools, achievement, and inequality: A seasonal perspective. Educational Evaluation and Policy Analysis , 23 (2), 171–191.
Graham, S., Berninger, V., & Fan, W. (2007). The structural relationship between writing attitude and writing achievement in first and third grade students. Contemporary Educational Psychology , 32 (3), 516–536. Liew, J., McTigue, E. M., Barrois, L., & Hughes, J. N. (2008). Adaptive and effortful control and academic self-efficacy beliefs on achievement: A longitudinal study of 1st through 3rd graders. Early Childhood Research Quarterly , 23 (4), 515–526.
Unrau, N., & Schlackman, J. (2006). Motivation and its relationship with reading achievement in an urban middle school. The Journal of Educational Research , 100 (2), 81–101.
LITCAMP EVALUATION IN SCHENECTADY CITY SCHOOL DISTRICT 17
Appendix A LitCamp Activities
Opening Campfire Students begin each day with a song and a community-building activity, play interactive games, and/or engage in writing activities as they are introduced to new words to support their vocabulary development. Read Aloud Through teacher-led interactive read-alouds, students strengthen their speaking, listening, reading, and writing skills by expressing their understanding of a story and connecting the text to the 7 strengths and their own lives. Bring the Text to Life Students enhance their understanding of the read-aloud text by performing, dancing, or drawing about important ideas discussed, including the characters’ thoughts and actions. Reading Power Students build literacy skills in teacher-led small- and whole-group activities that connect the read-aloud texts to higher standards, sometimes with the support of graphic organizers. Bunk Time Students have the opportunity to choose their own book and build stamina as they read independently. This time also includes opportunities to share what they are reading with a partner or the teacher. Community Lit Students continue to build their literacy skills, especially speaking and listening, and build a sense of community by participating in group discussions or games about the 7 strengths. Writing Power Students engage in a writing activity to extend their learning and make personal connections to the read-aloud text and the 7 strengths. Closing Campfire Students end each day with a vocabulary activity to reinforce the day’s learning and with opportunities for self-reflection and for teachers to praise students for their achievements.
18
SCHOLASTIC RESEARCH & VALIDATION
Appendix B Evaluation Methodology
Scholastic Research & Validation partnered with RMC Research to conduct a third-party evaluation of LitCamp in Schenectady City School District during summer 2018. Four sites participated in the evaluation, implementing LitCamp as the primary literacy component for approximately two hours per day as part of the district’s 4-week Summer Enrichment Program. The Summer Enrichment Program ran from Monday through Friday for 10 hours per day. LitCamp served 971 students who spanned seven grade levels (kindergarten through Grade 6). Evaluation Questions The evaluation used a mixed-methods design that involved quantitative data collected through student and teacher surveys and district reading achievement data and qualitative data collected through interviews, focus groups, and observations. Exhibit B1 presents the evaluation questions and the data sources and analyses used to answer each question.
LITCAMP EVALUATION IN SCHENECTADY CITY SCHOOL DISTRICT 19
Exhibit B1 Evaluation Questions
Evaluation Question
Data Sources
Focus and Analyses
1. How did students’ reading knowledge change as a result of their exposure to LitCamp?
• Student demographics
Analysis of covariance models were used to assess change in student assessment outcomes, controlling for pre-LitCamp scores and student demographic characteristics. Analysis of covariance models were used to assess change in student outcomes, controlling for pre-LitCamp scores and student demographic characteristics. Descriptive analyses were used to assess teachers’ perceptions of change in students’ attitudes and reading behaviors. Focus group and interview data were analyzed qualitatively and synthesized across teachers or family members to provide more in-depth information around the effect of LitCamp on students. Descriptive analyses were used to assess teachers’ perceptions of the program and change in students’ attitudes and reading behaviors. Teacher focus group data were analyzed qualitatively to pull out key themes and contextual background.
• Student assessment scores
2. How did students’ reading behaviors and attitudes toward reading change as a result of their exposure to LitCamp?
• Student demographics
• Student surveys
• Teacher surveys
• Teacher focus groups
• Student and parent interviews
• Teacher surveys
3. How did teachers perceive LitCamp?
• Teacher focus groups
• Teacher surveys
4. What did LitCamp
Classroom observation notes and teacher focus group data were analyzed qualitatively to pull out key themes and contextual background.
implementation look like in practice?
• Classroom observations
• Teacher focus groups
SCHOLASTIC RESEARCH & VALIDATION 20
Data Sources RMC Research collected data from the following sources:
• Student assessment data . The district provided RMC Research with reading achievement data from district-administered student assessments for students who participated in LitCamp and a district-identified statistically similar comparison group of students who did not participate in LitCamp. • Student demographics. The district provided RMC Research with demographic information for all enrolled LitCamp students, including EL status, special education designation, race and ethnicity, and gender. The district also provided demographic data for all comparison students. • Student surveys. RMC Research administered a student survey to all LitCamp students in Grades 3–5 at two time points: the beginning of LitCamp (baseline) and the end of LitCamp (follow-up). The survey asked questions related to noncognitive factors in students such as growth mindset, attitudes toward reading, the 7 strengths (i.e., belonging, curiosity, friendship, kindness, confidence, courage, hope), and included questions about students’ experiences at LitCamp. • Teacher surveys. RMC Research administered a beginning-of-session and end-of-session survey to all teachers. The baseline survey asked questions about their background and their expectations for and initial impressions of LitCamp. The end-of-session survey asked questions about their perceptions of the LitCamp program, its effectiveness at developing literacy and social-emotional skills in students, its effectiveness in cultivating students’ interest in reading, and students’ engagement with the program. • Classroom observations. RMC Research observed LitCamp instruction across multiple grades at each of the four sites to gather data on what LitCamp implementation looked like in practice. • Focus groups. RMC Research conducted a teacher focus group at each of the four Summer Enrichment Program sites to gather more in-depth information from teachers pertaining to their perceptions of LitCamp materials and activities, their perceptions of the program’s impact on students’ reading behaviors and social-emotional behaviors, and student engagement and attitudes. • Family interviews. RMC Research conducted a total of four interviews at two Summer Enrichment Program sites with LitCamp students and their families to gain deeper insight into the effect that LitCamp had on those students.
LITCAMP EVALUATION IN SCHENECTADY CITY SCHOOL DISTRICT 21
Target Sample Student assessment. RMC Research targeted all students participating in LitCamp for the student assessment sample. The district provided enrollment and demographic data for 971 students in kindergarten through Grade 6 and included spring 2018 assessment scores for 884 of those students. The district created a statistically similar matched comparison group for a subset of LitCamp students who had both a spring and fall 2018 AIMSweb assessment score ( n = 266). 16 The final assessment sample included 266 LitCamp students (27% of all 971 enrolled LitCamp students) and 266 comparison students. Comparison students were statistically similar to LitCamp students on demographic characteristics and spring 2018 assessment scores. Student surveys. RMC Research targeted a subset of grades across all four sites for the student survey sample, and the district provided enrollment data for 395 students in those grades (Grades 3–5). The final targeted sample included 358 students in Grades 3–5 due to 37 students who were enrolled but did not attend.
Teacher surveys. The targeted teacher survey sample included all 80 teachers who taught LitCamp to students in kindergarten through Grade 6.
Teacher classroom observation and focus groups. RMC Research invited kindergarten through Grade 5 teachers to participate in a classroom observation and focus group at each site with the goal of recruiting four teachers per site for classroom observations and six to eight teachers per site for focus groups. 17 Family interviews . RMC Research invited a small subsample of students ( n = 4) and their families (i.e., parents and other LitCamp siblings) from two of four Summer Enrichment Program sites to participate in face-to-face interviews. Site leaders nominated the families to participate in the family interviews. Data Collection Procedures RMC Research utilized a combination of surveys, classroom observations, focus groups, and interviews to gather data for the evaluation. In addition, RMC Research worked with district staff to obtain student demographic and reading assessment data. RMC Research provided a designated leader at each site with detailed instructions for the survey administration. At the beginning of each survey administration period, the site leaders distributed student survey materials to teachers of Grades 3–5 and teacher surveys to each teacher. At the end of each survey administration period the site leaders collected the materials from teachers and returned them to RMC Research using the prepaid shipping label provided by RMC Research.
16 The district changed assessments between spring and fall 2018, and the district only tested approximately one-third of all Summer Enrichment Program students on AIMSweb in fall 2018. Tested students were those who scored in the lowest tier in spring 2018. 17 The study did not target Grade 6 because few Grade 6 students participated in the Summer Enrichment Program and only at one site.
SCHOLASTIC RESEARCH & VALIDATION 22
Page i Page ii Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Page 9 Page 10 Page 11 Page 12 Page 13 Page 14 Page 15 Page 16 Page 17 Page 18 Page 19 Page 20 Page 21 Page 22 Page 23 Page 24 Page 25 Page 26 Page 27 Page 28 Page 29 Page 30 Page 31 Page 32Made with FlippingBook - Online catalogs