After-School Program Effects on Behavior: Results from

the 21st Century Community Learning Centers Program National Evaluation

December 21, 2006

Susanne James-Burdumy
Mark Dynarski
John Deke

Mathematica Policy Research, Inc.

ABSTRACT

This paper presents evidence on after-school programs effects on behavior from the national
evaluation of the U.S. Department of Educations 21st Century Community Learning Centers
after-school program. Findings come from both of the studys components: (1) an elementary
school component based on random assignment of 2,308 students in 12 school districts and (2) a
middle school component based on a matched comparison design including 4,264 students in 32
districts. Key findings include higher levels of negative behavior for elementary students and
some evidence of higher levels of negative behaviors for middle school students.

1
 I. INTRODUCTION

The number of after-school programs has grown rapidly in recent years, spurred by growing

employment rates of mothers, pressure to increase academic achievement, and concerns about

risks to children who are unsupervised during after-school hours. After-school programs have

been hypothesized to improve child and youth behavioral outcomes, but evidence on whether

they do is mixed. Some studies have reported that after-school programs reduce negative

behaviors (see Tierney et al. (1995) and Marshall et al. (1997)). Other studies have shown no

effect of after-school programs on behaviors (see Baker and Witt (1996) and the National

Institute of Child Health and Human Development Early Child Care Research Network study

(2004)). Some studies report that after-school programs or after-school care increased negative

behaviors (Massachusetts 2020 and Boston Public Schools (2004); Johnson and Donley (1999);

Vandell and Corasaniti (1988); Baker et al. (2005)).

In 1999, Mathematica Policy Research, Inc. (MPR), began a national evaluation of the U.S.

Department of Educations (ED) 21st Century Community Learning Centers program. It was the

first multi-state evaluation of after school programs to use an experimental design. This paper

describes findings from that evaluation related to behavior.1

a. The 21st Century Community Learning Centers Program

After-school programs in elementary schools were generally open for three hours after

school four or five days a week. Most centers offered homework assistance and academic

activities, such as teaching or tutoring. Nearly all centers offered recreational opportunities

ranging from unstructured free time to organized sports. Programs also offered enrichment

activities such as dance, drama, and music, as well as workshops on developmental topics such

as building leadership skills and resolving conflicts with peers.

2
 Elementary students in the treatment group attended the program an average of 81 days

during the two years of the study (49 days in the first year and 32 days in the second year).

Average attendance in the second year was lower because a quarter of students did not have

access to the program in the second year because they changed schools and their new school did

not have a 21st Century center, and because 47 percent of students who could have attended a

program in the second year did not attend.

Middle school programs shared many features with elementary school programs. About

80% of middle school centers offered homework sessions and 60% offered other types of

academic assistance, such as additional help in language arts or mathematics. Students typically

could choose from a variety of recreational and enrichment activities, such as free time in the

gym, board games, table tennis, computer lab, and arts and crafts.

Middle school students in the treatment group attended the program an average of 42 days

during the two years of the study (33 days in the first year and 9 days in the second year).

Average attendance in the second year was lower because 59% of students did not have access to

the program in the second year and because 53% of students who had access to the program in

the second year did not attend.

b. Evaluation Design

Elementary Schools. The evaluation identified two cohorts of 21st Century elementary-

school centers that had more applicants than slots and randomly assigned students to be able to

attend programs (the treatment group) or not (the control group). The control group could

participate in other after-school programs such as those operated by community organizations or

churches, but data indicate that the majority were at home after school with a parent or relative.

In total, 2,308 elementary students who were eligible for and interested in attending a 21st

3
Century center were randomly assigned either to the 21st Century treatment group (1,258

students) or to a control group (1,050 students). Random assignment was by center.

Middle Schools. For the middle school study, a nationally-representative sample of thirty-

five grantees that operated centers serving middle school students were selected at random from

16 strata, which were structured to represent geographic region and urban and rural areas.

Thirty-four districts agreed to participate, and, ultimately, 32 school districts and 61 schools in

these districts participated in the study (delays in baseline data collection led to dropping two

districts). Most middle-school programs were not oversubscribed, therefore, a comparison

design was used rather than random assignment. The comparison-group design tempers the

studys ability to attribute measured effects to the 21st Century program alone. Students

attending 21st Century centers during a one-month window in fall 2000 formed the participant

group. Propensity-score matching techniques (Rosenbaum and Rubin 1983) were used to

identify similar students not attending centers, either from the same schools in which centers

were located or from other schools in the district that did not operate 21st Century centers. In

total, 4,264 students (1,782 participants and 2,482 comparison students) were included in this

component of the study.

c. Data

The evaluation collected data on a wide array of outcomes, including grades, test scores,

classroom behavior and effort, absences, suspensions, location and supervision after school,

social development, parental involvement, negative behavior, and feelings of safety after school.

The evaluations data sources include questionnaires completed by students, parents, teachers,

principals, and program staff, as well as reading tests, school records, center attendance records,

and site visits. Baseline (fall of the school year), first follow-up (spring of the school year), and

4
second follow-up data (spring of the next school year) were collected. This paper focuses on

second follow-up outcomes. Second follow-up response rates were high, ranging from 76% for

reading test scores to 88% for the student survey.

Baseline data provide a measure of the similarity of the treatment and control groups in the

elementary study and the participant and comparison groups in the middle school study. Overall,

the elementary and middle school treatment and control/comparison samples were similar. Some

differences in parent-reported data were observed between the treatment and comparison groups

in the middle school sample, which did not use parent-reported data in its propensity-scoring

models (not collecting parent data reduced data-collection costs because data were collected only

from parents whose children ultimately were selected for the comparison group, which is a

smaller group than parents of children in the potential comparison group). A larger proportion of

participants than comparison students lived in households with low income and fewer

participants parents both held full-time jobs. These differences underscore the importance of

adjusting for student characteristics when estimating impacts.

d. Analysis Methods

Elementary Schools. The study estimated an impact for each of the 12 sites in the study and

calculated an overall impact as a simple average of the 12 site impacts. Regression models

included a set of baseline covariates to adjust for initial differences. Impacts for elementary

school participants also were estimated using two-stage least squares, with initial assignment to

treatment as an instrument for program participation.

Middle Schools. The study estimated impacts by regressing second follow-up outcomes on

an indicator of whether sample members were in the participant or comparison group, as well as

a set of baseline covariates. Baseline measures of outcome variables were included if available.

5
 II. RESULTS

a. Elementary Schools

The study found evidence that after-school programs contributed to behavior problems

(Table 1). For example, teachers reported calling parents about behavior problems for 28% of

treatment-group students and 23% of control-group students (effect size of 0.12). Twenty-two

percent of treatment-group students reported that they were disciplined for behavior compared to

17% of control-group students (effect size of 0.16). Twelve percent of treatment-group students

were suspended from school, compared to 8% of control-group students (effect size of 0.16).2

Results in the first follow-up were similar but not statistically significant.

Impacts on participants (not shown) were similar to the intent-to-treat impacts. For

participants, the impact on suspensions had a higher level of statistical significance than the

intent-to-treat model.

A composite variable based on teacher reports of discipline problems was significantly

higher for treatment-group students relative to control-group students, as was a composite

variable based on student reports of discipline problems. No individual impacts were observed

on the extent to which teachers reported disciplining the child for misbehaving, sending the child

to the office for misbehaving, or giving the child detention.

An analysis of six subgroups found that impacts on negative behavior were concentrated

among boys and students with high baseline levels of disciplinary problems. See James-

Burdumy et al. (2005) for details of this analysis.

b. Middle Schools

6
 Findings for middle-school students corroborate evidence of increased negative behavior,

though findings are more mixed than for elementary-school programs (Table 2). Program

participants had higher values on a negative behavior composite variable, and were more likely

to break things on purpose and take illegal drugs (such as cocaine, ecstasy, or LSD). Other

outcomes showed few differences between participant and comparison groups.

III. DISCUSSION

Hypotheses about why students who attended after-school programs were more likely to

misbehave during regular school include (1) students may be more fatigued and acting out

because they are spending more time in school, (2) students may be influenced negatively by

peers with whom they are spending time during the after-school program, or (3) students could

be misbehaving more because programs tolerated behavior for which students would be

disciplined during regular school. Data were not available to investigate the first and second

hypotheses above but some data were available to examine the third hypothesis.

If students behaved during school in ways that were tolerated in their after-school program

but not in school, they would likely experience more disciplinary actions. The study had data

about students perceptions of the disciplinary environment in school and in the after-school

program, based on questions about how true students felt the following statements were: when

kids break the rules at my school, they are disciplined and there are too many rules to follow in

my school. Corresponding questions were asked about the after-school program. Responses to

these questions showed that students who reported attending an after-school program were

significantly more likely to report that their school had a stricter disciplinary environment than

their after-school program. Sixty-five percent of these students thought it was very true that

7
kids are disciplined when they break rules in school, compared to 59% of students reporting this

about their after-school programs (p < 0.05). Similarly, 31% of these students reported it to be

very true that there are too many rules to follow in school, compared to 25% of students who

reported this about their after-school programs (p < 0.05).

IV. CONCLUSION

The evidence indicates that for elementary-school students, 21st Century after-school

programs increased negative behaviors. Subgroup impacts indicated that nearly all the negative

behavior could be traced to boys (behavior impacts for girls were close to zero and statistically

insignificant), and to students who had a higher level of disciplinary problems at baseline, which

may provide some insights about the pathways of the behavior problems. For middle school

students, there were increases in some negative behaviors, with a composite variable for five

negative behaviors being higher for the program group in both years. The increases were not

large but were statistically significant.

Additional analyses suggest that differences in disciplinary policies between after-school

programs and the school day may explain some of the increase in behavior problems. Future

studies would benefit from the inclusion of survey questions designed to address other

hypotheses advanced here.

8
 REFERENCES
Baker, D., and P. Witt. Evaluation of the Impact of Two After-School Recreation Programs.
Journal of Park and Recreation Administration, 14(3), 1996, 23-44.

Baker, M., J. Gruber, and K. Milligan. Universal Childcare, Maternal Labor Supply, and
Family Well-Being. NBER Working Paper No. 11832. Available online at
http://www.econ.ubc.ca/kevinmil/research/childcare.htm, 2005.

DeAngelis, K., and R. Rossi. Schools Serving Family Needs: Extended-Day Programs in Public
and Private Schools. National Center for Education Statistics Issue Brief (NCES 97-590).
Washington, DC: National Center for Education Statistics, 1997.

Dynarski, M., M. Moore, J. Mullens, P. Gleason, S. James-Burdumy, L. Rosenberg, C. Pistorino,

T. Silva, J. Deke, W. Mansfield, S. Heaviside, and D. Levy. When Schools Stay Open Late:
The National Evaluation of the 21st Century Community Learning Centers Program, First-
Year Findings. Report submitted to the U.S. Department of Education. Princeton, NJ:
Mathematica Policy Research, Inc., 2003.

James-Burdumy, S., M. Dynarski, M. Moore, J. Deke, W. Mansfield, and C. Pistorino. When

Schools Stay Open Late: The National Evaluation of the 21st Century Community Learning
Centers Program: Final Report. Report submitted to the U.S. Department of Education.
Princeton, NJ: Mathematica Policy Research, Inc., 2005.

Johnson, J., and J. Donley. Support Our Students (SOS) Program: 1998-99. Raleigh, NC: 4-H
Youth Development NC Cooperative Extension Service. 1999.

Marshall, N., C. Garcia Coll, F. Marx, K. McCartney, N. Keefe, and J. Ruh. After-School Time
and Childrens Behavioral Adjustment. Merrill-Palmer Quarterly, 43(3), July 1997, 497-
514.

Massachusetts 2020 and Boston Public Schools. The Transition to Success Pilot Project.
Boston, MA: Massachusetts 2020, 2004.

National Institute of Child Health and Human Development Early Child Care Research Network.
Are Child Developmental Outcomes Related to Before- and After-School Care
Arrangements? Child Development, 75, 2004, 280295.

Rosenbaum, P. & Rubin, D. (1983). The central role of the propensity score in observational
studies for causal effects. Biometrika, 70.

Tierney, J., J. Baldwin Grossman, and N. Resch. Making a Difference: An Impact Study of Big
Brothers/Big Sisters. Philadelphia, PA: Public/Private Ventures, 1995.

Vandell, D. L., and M. A. Corasaniti. The Relation Between Third-Graders After-School Care
and Social, Academic, and Emotional Functioning. Child Development, 59, 1988, 868875.

9
 Table 1
Impacts on Behavioral Outcomes, Elementary School Centers, Year 2

Treatment Control Estimated

Outcome Group Group Impact

Percentage of Students Who Rate Themselves as Excellent on

Sticking to What They Believe in, Even if Their Friends Dont
Agree 50.0 52.1 -2.1
Mean Number of Days Student Was:
Absent 8.1 8.4 -0.2
Late 4.4 4.3 0.1
Percentage of Students Whose Teachers Report Doing the
Following Two or More Times:
Disciplining the child for misbehaving 47.0 43.2 3.8
Sending child to the office for misbehaving 13.6 12.4 1.2
Giving child detention 20.8 17.5 3.4
Calling parents about childs behavior 28.1 23.1 5.1**
Percentage of Students Who Report the Following Happens
Some or A Lot:
Student has to miss recess or sit in the hall 22.4 16.9 5.5**
Parents have to come to school about problem 22.3 16.8 5.6**
Student-Reported Discipline Problem Composite (Mean) 1.7 1.6 0.1**
Teacher-Reported Discipline Problem Composite (Mean) 1.8 1.7 0.1**
Student-Reported Negative Behavior Composite (Mean) 1.7 1.7 0.0
Percentage of Students Who Were Suspended During Most Recent
School Year 11.5 7.5 4.1**
Sample Size 1,055 880

SOURCE: Student Survey, School Records, Teacher Survey.

NOTE: This table shows intent-to-treat estimates which use the full treatment and control groups.
The percentages and mean values of outcomes for treatment and control students have been
regression-adjusted for baseline differences between the groups. The control variables in the
regression included student characteristics such as indicators of students demographic
characteristics, students baseline test scores, and school attendance. Weights are used to adjust
impact estimates for nonresponse.
**Significantly different from zero at the .05 significance level, two-tailed test.
***Significantly different from zero at the .01 significance level, two-tailed test.

10
 Table 2
Outcome Differences in Behavior, Middle School Centers, Year 2

Center Comparison
Outcome Participants Group Difference

Percentage of Students Who Rate Themselves as Excellent on Sticking to What They

Believe in, Even if Their Friends Dont Agree 41.5 43.2 -1.7

Mean Number of Days Student Was:

Absent 9.0 10.0 -1.0**
Late 6.2 5.4 0.8
Percentage of Students Whose Teachers Report Doing the Following Two or More
Times:
Disciplining the child for misbehaving 20.0 19.9 0.1
Sending the child to the office for misbehaving 6.8 6.8 0.0
Giving the child detention 9.9 10.0 -0.1
Calling parents about the childs behavior 9.5 8.9 0.7

Student-Reported Discipline Problem Composite (Mean) 1.4 1.4 0.0

Teacher-Reported Discipline Problem Composite (Mean) 1.4 1.4 0.0

Percentage of Students Who Were Suspended During 2001-2002 School Year 21.9 21.7 0.2

Percentage of Students Who Report That They Do the Following Some or A Lot:
Break something on purpose 10.4 8.0 2.4**
Punch or hit someone 22.4 19.7 2.7
Steal from a store 4.9 4.0 0.9
Sell illegal drugs 1.4 1.8 -0.3
Get arrested or detained by police 3.3 3.1 0.2

Student-Reported Negative Behavior Composite (Mean) 1.56 1.53 0.03**

Percentage of Students Who Report That They Did the Following Some or A Lot:
Smoke cigarettes 4.7 4.1 0.6
Have at least one alcoholic drink 9.8 9.0 0.8
Smoke marijuana 4.8 4.3 0.5
Took illegal drugs such as cocaine, ecstasy, or LSD 0.8 0.2 0.6***

Student-Reported Tobacco, Alcohol, and Drug Use Composite (Mean) 1.14 1.12 0.02
Sample Size 1,633 2,198

SOURCE: Student Survey, Teacher Survey, Parent Survey, School Records.

NOTE: The percentages and mean values of outcomes for participants and comparison group members have been
regression-adjusted for baseline differences between the groups. The control variables in the regressions include
student characteristics such as indicators of students demographic characteristics, students baseline test scores,
attendance, disciplinary problems, and self-reported grades. Due to rounding, estimated outcome differences shown
in the table do not always equal the difference between center participants and the comparison group. Weights are
used to adjust estimates for nonresponse. Variances are estimated using SUDAAN to account for the statistical
sampling design. Appendix A of James-Burdumy et al. (2005) describes how weights were constructed, and
Appendix B describes methods used to estimate outcome differences.

**Significantly different from zero at the .05 significance level, two-tailed test.
***Significantly different from zero at the .01 significance level, two-tailed test.

11
 NOTES
1
Dynarski et al. (2003) and James-Burdumy et al. (2005) provide additional information
about the evaluations design and analytic approach, and present results from a variety of
sensitivity analyses.
2
Discussions with program directors indicated that students were not likely to be suspended
from school because of their behavior during the after-school program, which means that
suspensions are related to negative behavior during the regular school day.

12