1. No category

Capturing positive outcomes for youth in diverse Out-of-School Time Programs
Why the Online Toolbox?
 Measures program quality and student outcomes
 Research-based scales with well-established reliability
and validity
 Easy to administer online surveys
 Choose respondents: Students, Program Staff, and/or
Classroom Teachers
 Program reports of site and program level data
 Online support for understanding program reports
afterschooloutcomes.org
Research Background of Measures

Youth self-reports assess program environmental features
shown to enhance youth outcomes:
o Experience with program activities
o Supportive relations with staff and peers

These student reports of program experience correlate highly
with observational data of program quality as assessed by the
Promising Practices Ratings System (PPRS; Rosenthal & Vandell,
1996).

Youth reports of program quality are linked to positive youth
outcomes (Kataoka & Vandell, 2013).
afterschooloutcomes.org
Research Background of Measures

The measures address 21st Century non-cognitive
competencies identified by the National Research Council:
o Personal and Social-Emotional Skills
o Social Competencies
o Academic Efficacy and Work Habits

Research links socio-emotional development to academic
gains (Durlak, Weissberg & Pachan, 2010).
afterschooloutcomes.org
Online Toolbox Surveys
Three web-based surveys:
1.
Student self-report (grades 3-12)
2.
Program staff reports of student skills and
behavior
3.
Classroom teacher reports of student skills and
behavior and academic outcomes for students
in their class who attend the afterschool
program
afterschooloutcomes.org
DESCRIPTION OF MEASURES
STUDENT SELF-REPORT
PROGRAM STAFF REPORTS &
CLASSROOM TEACHER REPORTS
Skill Development
 Work Habits
Students’ Skill Development
 Reading Efficacy

Work Habits
 Math Efficacy

Task Persistence
 Science Efficacy

Academic Performance
Positive Behavior Change
Students Behaviors
(classroom teachers only)
 Social Competencies

Aggressive Behavior with Peers
 Misconduct

Prosocial Behavior with Peers

Social Skills
Program Quality
 Relationships with Staff
 Interest in Program Activities
 Relationships with Peers
Also available STEM-related measures of
Student’s Attitudes and Beliefs
 Science Interest
 Science Career
 View of Future
afterschooloutcomes.org
Student Survey Self-Report
Measures of Skill Development
# Items
Sample Item
4-point Rating Response
Work Habits
6
I finish my work on time.
1 = Not at all true; 4 = Really true
Reading Efficacy
4
I expect to do well in reading.
1 = Not at all true; 4 = Really true
Math Efficacy
4
I am interested in math.
1 = Not at all true; 4 = Really true
Science Efficacy
4
I am interested in science.
1 = Not at all true; 4 = Really true
afterschooloutcomes.org
Student Survey Self-Report
Measures of Behavior
# Items
Sample Item
4-point Rating Response
Social
Competencies
7
I work well with other kids.
1 = Poor; 4 = Excellent
Misconduct
9
I have gotten into a fight at
school.
0 = Never; 3 = More than once a week
afterschooloutcomes.org
Student Survey Self-Report
Measures of Attitudes and Beliefs, Student Self-Report
# Items
Sample Item
4-point Rating Response
Science Interest
24
I get excited about learning
new discoveries or inventions.
1 = Not at all true; 4 = Really true
Science Career
12
I will get a job in a sciencerelated area.
1 = Not at all true; 4 = Really true
View of Future
7
I will go to college.
1 = Not at all true; 4 = Really true
afterschooloutcomes.org
Student Survey Self-Report
Measures of Quality of Program Experience
# Items
Sample Item
4-point Rating Response
Relationships
with Staff
6
I trust the teachers here.
1 = Not at all true; 4 = Really true
Relationships
with Peers
5
I get to know other kids really
well here.
1 = Not at all true; 4 = Really true
Program
Activities
5
I like the activities here.
1 = Not at all true; 4 = Really true
afterschooloutcomes.org
The Promising Practices Ratings System*
The PPRS is an observational tool that assesses eight
aspects of program quality:
 Supportive relations with
adults
 Opportunities for cognitive
growth
 Supportive relations with
peers

 Student engagement in
activities
 Setting chaos
 Mastery orientation
 Staff overcontrol
* Vandell et al. (March, 2005)
Appropriate program
structure
The Promising Practices Ratings System
 The PPRS eight program processes have been
linked to positive social and academic outcomes
for children and adolescents. *
 The PPRS can be used to supplement the Online
Toolbox student outcome data with observational
data of program quality, or it can be used by itself.
*(Eccles & Gootman, 2002; Pierce et al., 1999; Vandell, Shumow, & Posner, 2005)
afterschooloutcomes.org
Implementation of
Online Toolbox
Online Toolbox Implementation
 Available for use by all Out-of-School Time
programs nationally
 Used statewide by afterschool and summer
learning programs in California since 2013
 Pilot implementation by 21st CCLC funded
programs in Utah during 2014-15
California Statewide Implementation
Funded by the California Department of Education After School Division

Free-of-Cost Opportunity
for California ASES and 21st
CLCC Programs

Over 1,000 sites selected
to participate in 2013-14

Up to 2,000 sites can
participate in 2014-15
afterschooloutcomes.org
California Statewide Implementation
Funded by the California Department of Education After School Division
Benefits of Participation:

Programs receive a report summarizing student outcome scores and
student reports of quality of program experience.

The report includes a summary of student scores for each
participating program site in addition to the aggregate scores for
their program.

Programs can compare scores over time and their program
aggregate score with the statewide results.

Results can be used to inform internal program quality improvement
and to report program outcomes to funders and other stakeholders.
afterschooloutcomes.org
How does it work?
GO TO WEBSITE
afterschooloutcomes.org
COMPLETE ONLINE
INTEREST FORM
STEP-BY-STEP
INSTRUCTIONS &
LINKS TO SURVEYS
Steps to Participate
Complete Interest Form
1.
•
If not California ASES or 21stCCLC program contact afterschool@uci.edu
2.
Complete Site List Template and Submit
3.
Receive notification by email of selected programs and sites
4.
Complete Participant Data Template and Submit
5.
Receive Survey Codes to access surveys through Secure Program
Portal Page
 Download Instructions on Administering Surveys
6.
Administer Surveys
7.
Program Portal Page: Use to retrieve survey codes, see tallies of
surveys collected at your program sites, change password, add
new participants
http://afterschooloutcomes.org/ases-surveys
INSTRUCTIONAL VIDEOS
Click on
Appropriate Link
for Instructional
Videos
AOMOT TEAM
a f t e r s c h o o l @ u c i .e d u
949-824-9225