Tapestry of Chautauqua County, New York How to Build Partnerships With School Districts to Utilize 100% Longitudinal Data in the Evaluation of Systems of Care and its Impact on School Outcomes Workshop presented at The Summer 2011 System of Care Community Training, Chicago, July 19-21, 2011 Mansoor A. F. Kazi, PhD (Lead Evaluator), Jon Anderson(System Developer), Joe Yelich (Principal Jamestown Senior High School), Anne Bartone (PhD student) Realist Evaluation Partnerships Mansoor A. F. Kazi, PhD, Director, Program Evaluation Center, School of Social Work, University at Buffalo mkazi@buffalo.edu Tapestry of Chautauqua County, New York Youth Offending Teams, Local Authorities. UK Based on Kazi, M. A. F. (2003) ‘Realist Evaluation in Practice’, London: Sage Main Objectives--How to: • •Build a sustainable collaborative partnership with school districts • Evaluate the impact of services in achieving the desired outcomes and to use evidence in real-time (local and national) in partnership with school districts • •Apply realist evaluation strategies in the evaluation of 100% natural samples in schools together with school district colleagues • •Use tools such as Binary Logistic Regression to help understand and predict what works and for whom Chautauqua Tapestry Chautauqua Tapestry Mayville, NY Welcome to Chautauqua County! On Your Mark! Set…Go! • Chautauqua County’s Department of Mental Hygiene applied on behalf of Chautauqua County for a Cooperative Agreement in 2008.  Where we started & What lead us to where we are today! Evolution of Evaluation in Chautauqua County  2006 Child Welfare initiated Solution Focus Training for all child welfare staff  2008 the impact of Solution Focus Training not known  Dr. Kazi was introduced to Child Welfare and an agreement was executed for evaluation of Solution Focus Practice  Evaluate Solution Focus Practice in relation to Federal CFSR Standards First Realist Evaluation in Chautauqua County  Found to have a positive impact on LOS in Institutional Placement  First Realist Evaluation in Chautauqua County Evolution of Evaluation in Chautauqua County  2008 Chautauqua County Department of Mental Hygiene application for Cooperative Agreement with SAMHSA  Evaluation Requirement  Sought Assistance of Dr. Kazi  Dr. Kazi became Lead Evaluator for project Moving Ahead…  Strong, Diverse Cross-system representation on Governance  Each initiative/partner interested in Evaluation to support & guide investment of resources  Collaboration and Cross-System Partnership allowed discussion to evolve, multisystem perspective  Cooperative Agreement funds have permitted Evaluation to be partner at the table  School’s were an immediate partner having rich data, but little evaluation built within the system concerning the impact of strategies for the target population Evaluation in Schools – Critical Elements  Involve school personnel at Governance Level  Start with a strong partner school  Ask for testimonials from the school  Ask the school partner to share their experience at Superintendents, Principals and other school meetings  Share school evaluation experience at other key community meetings  Engage each school face-to-face  Let individual schools lead the evaluation process to correlate with their key interest areas Result Year Three  Twenty Partner agencies involved in the Local Evaluation under the supervision of Dr. Kazi  Eleven School Districts Highly engaged, or beginning engagement with Dr. Kazi  Community is developing a more comprehensive understanding of impact of service for children and families Two Key Components of Evaluation • Participate and facilitate SAMHSA’s national evaluation • Implement the local evaluation Local Evaluation • • • Realist evaluation (Kazi, 2003) to evaluate what interventions work and in what circumstances Build partnership and integrate evaluation strategy in practice with each agency and the wraparound services Enable youth and family participation and perspective within the evaluation strategy How to Conduct a Local Evaluation: What Interventions work & in what circumstances  A combination of efficacy research & epidemiology traditions  Management Information System (MIS) Data routinely collected but typically not used for evaluation in agencies  Investigate interrelationships between outcomes, client demographics, client circumstances, & services provided  Methods such as binary logistic regression can predict the likelihood of effectiveness of an intervention in given circumstances  Use findings at regular intervals to better target and develop services Local Evaluation Strategy Participants              Local Evaluation of nearly 29 agencies & list growing 100% evaluation of all services for children and families in Chautauqua County Within the 100%, evaluate the difference made by System of Care, Wraparound, etc. 11 school districts, i.e. Jamestown, Dunkirk, Chautauqua Lake, Silver Creek, Pine Valley, Westfield, Sherman, Frewsburg, Southwestern, Ripley, Forestville Chautauqua County Department of Mental Hygiene Chautauqua Opportunities Inc. Chautauqua County Department of Probation Family Service of the Chautauqua Region Gustavus Adolphus Family Services Gustavus Adolphus Learning Center Chautauqua County Department of Social Services WCA Chemical Dependency Program & mental health clinics Family court Local (Continued), State and International Participants                    The Resource Center Striders Infinity Visual and Performing Arts CASAC Compeer Chautauqua Works BOCES (Hewes Center) Jamestown Community Learning Council (early years) Chautauqua County Health Department County Mental Health Clinics and several others in Rensselaer County, NY Averill Park school district in Rensselaer County, NY Local Authorities in Scotland (looked after children) Moray Council, Aberlour Trust in Scotland Moray Youth Justice Team Local Authorities in England (looked after Children) Manchester Youth Offending Team Liverpool Youth Offending Team Finland: Helsinki Social Services Department Viewpoint international: UK & Australia (surveys of children) EXAMPLES of Realist Evaluation • • • • • • • Jamestown, Chautauqua Lake, Dunkirk, School Districts (Chautauqua County) and Averill Park School District (Rensselaer County) 2008/09 baseline and comparisons with 2009/10 and each marking period in the current year Outcomes: average school grades, state tests, discipline, attendance, drop out rates Demographics: ethnicity, gender, lunch status, IEP, etc. Interventions: school based interventions, summer program, mental health and other services 100% school data plus 100% agency data from participating agencies What works and for whom in achieving school, agency and system of care outcomes Data analysis and utilitization • • • • • • • Single system design with each youth and one group pretest postest design repeated at every marking period Comparison of outcomes between baseline and subsequent periods Comparisons between those receiving and not receiving interventions Investigation of patterns between outcomes, demographics and interventions Binary logistic regression to identify predictors at every marking period Data analysis carried out in partnership with schools and agencies Utilization of evaluation findings to develop and improve services for children and families at regular intervals Examples of data and findings • Next phase of workshop is to demonstrate individual databases from the school districts • Demonstration of how to use existing data and to undertake data analysis • How to build partnerships in data collection, analysis, and dissemination Conclusion • Realist evaluation strategies help to utilize the available data and to conduct deeper analysis • Explanations for differences between those that improve on an outcome and those that do not • Identify patterns in the data with a prediction of the odds of achieving a given outcome in particular circumstances • Repeatedly investigate changes in outcomes, intervention & the contexts with schools, agencies, and communities