Evaluation Highlights for Seneca Family of Agencies’
Investing in Innovation (i3) Grant
In 2014, Seneca Family of Agencies (Seneca) was awarded a grant from the Investing in Innovations (i3) Fund, sponsored by the U.S. Department of Education, to support the implementation and evaluation of the Unconditional Education (UE) model. The UE model promotes an innovative, multi-tiered framework that integrates academic, behavioral, and social emotional supports for students, while explicitly addressing and improving the overall school culture and climate for the entire school community. UE was launched during the 2014-2015 school year in partnership with seven schools in Oakland and San Francisco. SRI International has been the external evaluator for this grant, and this summary is pulled from a draft of their evaluation report covering the first year of implementation (2014-2015 school year).
The i3 Fund requires a rigorous evaluation design. SRI International has worked with Seneca to utilize a quasi-experimental design to compare outcomes for students at the participating schools with matched comparison students within the same district. The outcomes listed below are considered long-term outcomes, and therefore it is expected to take at least two to three years of implementation to realize statistically significant effects. Evaluation Questions Include:
After one, two, and three year(s) of implementation, compared to similar students in schools that did not participate in UE, did students who attended schools that participated in the UE program demonstrate:
- higher scores in English Language Arts achievement
- higher scores in Mathematics achievement
- better attendance
- lower suspension/expulsion rates
SFUSD Limitations: SFUSD data have limited power to detect significant relationships due to the small sample sizes, particularly for subgroup analysis. This is due in large part to SFUSD’s requirement to attain active consent in order to include student data in any research study, and the challenge in getting active consent forms signed and returned for all students. An additional limitation is that the district was not willing to share Free and Reduced Priced Lunch status for comparison students. As a result, this crucial factor was not included when matching UE students to other students within the district. Due to these limitations, we will not include analysis for SFUSD students, specifically, in this summary.
Effect Size: While some outcomes were found to be statistically significant, a number of outcomes registered a positive effect without reaching the level of statistical significance. In this summary, we will share outcomes with an effect size larger than .20 and/or statistically significant effects, acknowledging that anything below a .20 is generally considered trivial..
COMPARISON DESIGN STUDY OUTCOME HIGHLIGHTS
The results for students overall were mixed, resulting in negligible effects when looking at Oakland and SFUSD students combined. However, when looking at mathematics specifically, it was found that students within Oakland schools experienced positive effects mathematics (.27), as did the overall, combined set of students (.17). Attendance rates were generally so high within both UE and comparison groups overall, and suspension rates generally so low within UE and comparison groups overall, that negligible effects were found. The impact on mathematics achievement of UE was found to be statistically significant for students within Oakland, and for the overall, combined set of students.
Students in Special Education
This is a particularly important subgroup because UE has an important focus on improving the academic achievement, inclusion, behavior, and social-emotional well-being of students with disabilities. In addition, the project was funded under the “Students with Disabilities” i3 priority. While the comparison design study did not find any statistically significant results in year one, students with disabilities did realize statistically significant growth in reading and comprehension skills as outlined in the supplemental evaluation on the next page.
African American Students
The results for African American students were mixed, resulting in negligible effects when looking at Oakland and SFUSD students combined. In Oakland, African American students experienced small to moderate positive effects in mathematics (.27).
The ability of the UE model to effectively serve Latino students is crucial, as nearly 80% of students receiving UE in Oakland schools are Latino. Overall, Latino students experienced positive effects in reading (.19) and mathematics (.23). When looking at Oakland students, specifically, Latino students experienced positive effects in reading (.22) and math (.29). The impact on ELA mathematics achievement of UE was found to be statistically significant for Latino Students within Oakland, and for the overall, combined set of Latino students.
English Language Learners
Similarly, the ability of the UE model to effectively serve English Language Learners (ELLs) is crucial, as many schools participating in the project serve high percentages of ELLs (nearly 50% of students in Oakland). Overall, ELLs experienced positive effects in reading (.34) and mathematics (.34). When looking at Oakland students, specifically, ELLs experienced positive effects in ELA (.32) and math (.36). The impact on ELA and math achievement of UE was found to be statistically significant for English Language Learners within Oakland, and for the overall, combined set of ELL students.
SUPPLEMENTAL EVALUATION FOR STUDENTS WITH DISABILITIES
Data for this supplemental evaluation came from the direct assessment of students with disabilities in the five participating Oakland schools. The assessment tool from the Special Education Elementary Longitudinal Study (SEELS) contains research versions of the Comprehensive Test of Phonological Processing (CTOPP), Woodcock-Johnson Tests of Achievement, the Student Self-Concept Scale, and the Attitudes Toward School Scale. The baseline (pre-) assessment was completed before the UE model was implemented, in the spring of 2014, and the post-assessment was implemented during the spring of 2015.
- Students with disabilities showed improvement on 9 of the 13 outcomes, with four of these showing statically significant improvement
- Within the Literacy measures, students showed statistically significant growth on word segmentation, passage comprehension, and two measures of oral reading fluency.
- No outcome measures for school attitudes or self-concept showed statistically significant improvement, although the positive effect sizes associated with social (.17) and overall self-concept (.24) were sizably different from zero