The
Evaluation Exchange
Volume IX, No. 3, Fall 2003
Issue
Topic: Evaluating Community-Based Initiatives
Promising Practices
Alternative Designs for Community-Based Research:
Pittsburgh’s Early Childhood Initiative
Stephen Bagnato, Robert Grom, and Leon Haynes describe an
evaluation design for Pittsburgh’s Early Childhood Initiative
that provides scientific rigor in a community setting.
Little
agreement exists about how evaluations of social intervention
programs should be conducted. Traditional social scientists
argue for the use of laboratory-based, control group, randomized
designs as the gold standard, but this approach lacks generalizability
to real-life settings. Alternative evaluation designs are
necessary to document the elements of intervention programs
that predict outcomes in natural community settings. Yet critics
charge alternative methods with a lack of experimental rigor.1
An evaluation approach known as authentic assessment and program
evaluation research meets the demand for rigor while addressing
the community setting context.
Pittsburgh’s
Early Childhood Initiative
In 1994, as part of the Early Childhood Initiative (ECI),
the Heinz Endowments organized the business, corporate, agency,
and foundation sectors in Pittsburgh to expand quality early
care and education programs and options for unserved children
in high-risk neighborhoods. The overarching mission of ECI
is to foster preschool and school success for children of
poverty, whose typical retention and special education placement
rates in kindergarten have ranged between 18% and 40%.
A consortium
of business, community, and foundation leaders designed the
goals, approach, and expected outcomes of ECI. This design
was based on seven core features of successful early childhood
programs for children at developmental risk that were identified
by Craig Ramey and Sharon Ramey in their article, Early Intervention
and Early Experience.2 The seven core features include: (1)
longitudinal interventions starting in infancy and monitored
through functional benchmarks; (2) intensive, comprehensive,
and individualized programs and supports; (3) integral parent
participation; (4) high program quality and frequent monitoring;
(5) direct child interventions; (6) community-directed programs
and integrated services; and (7) follow-through of child and
family supports and program evaluation into the primary grades.
Several
Pittsburgh urban neighborhoods have participated in this collaboratively
designed and privately funded joint venture. Braddock’s
4 Kids Early Childhood Initiative and the Wilkinsburg ECI
are two of the most distinctive of these community-driven
ventures. A community leadership council established in Braddock
forged a relationship between Woodland Hills School District,
Head Start, and various formal and informal resources in the
community (e.g., churches, U.S. Department of Housing and
Urban Development community councils, and local hospital networks)
to link services for children and families. In Wilkinsburg,
Hosanna House, a broad community service center, incorporated
family support programs as central features of their early
care and education programs. In fact, these communities have
lead efforts to incorporate the School Readiness Group, a
nonprofit early childhood consortium, in order to harness
the influence of cross-community partners to advocate for
government, foundation, and agency funding.
SPECS
Authentic Program Evaluation Research Model
In 1996 the Heinz Endowments and the ECI Management Council,
composed of business, corporate, foundation, and community
members, selected an interdisciplinary research team from
Children’s Hospital of Pittsburgh and the UCLID Center
at the University of Pittsburgh known as SPECS (Scaling Progress
in Early Childhood Settings), as the winners of a national
competition to conduct ECI’s longitudinal evaluation.
SPECS’
evaluation approach—authentic assessment and program
evaluation research—helps community-based programs demonstrate
“how good they are at what they do.” It has been
validated in the field through evidence-based research conducted
through “natural experiments” in real-life community
settings rather than laboratory settings.3 SPECS’ strategies
are unique and effective because they:
Use
a collaborative research model with community partners for
the formative and summative research phases.
Ask whether the program works in a natural setting rather
than a laboratory setting.
Assess all children, families, and programs in the study without
exclusions.
Apply the developmentally appropriate quality guidelines of
the National Association for the Education of Young Children,
the Division for Early Childhood, Council for Exceptional
Children, and the Head Start Performance Standards.
Do not use traditional “tabletop testing” and
remove the child, teachers, or parents from their natural
situation or “developmental ecology.”
Rely on ongoing observations from consistent caregivers in
the child’s life.
Sample skills within the preschool’s developmental curriculum
that are teachable and predictive of future kindergarten success.
Offer ongoing feedback to teachers, parents, and the community
about children’s learning and needed program refinements.
Operationalize a longitudinal repeated-measures design using
HLM and path analysis strategies.4
SPECS’ research methods track progress and interrelationships
among multiple factors like children’s development (e.g.,
basic concepts, literacy, social skills, and self-control
behaviors), parenting and family strengths, the standards
and “best practices” of early childhood programs,
and neighborhood resources and interagency partnerships in
systems reform efforts.
The
Results of the Early Childhood Initiative
The SPECS evaluation team carefully tracked the progress of
1,350 enrolled children between 1997 and 2003. The team observed
and profiled progress three times each year, focusing on thinking,
language, early literacy, social, behavioral, and play skills.
They regularly provided feedback to teachers and parents to
guide their teaching and care. They also conducted program
quality evaluations in 25 programs in nine Pittsburgh neighborhoods
(Braddock, Wilkinsburg, Sto-Rox, East Liberty, South Side,
Highlands, Hill District, Homewood, and Steel Valley).
SPECS’
research on ECI’s impact showed major outcomes in four
areas (for details see the box):
Children
beat the odds and learned early skills for school success.
Mentored programs achieved stringent quality standards.
With teachers’ help, parents learned new ways to nurture
their children’s development.
Communities proved their leadership and made their programs
successful.
To download the full SPECS report or executive summary go
to www.uclid.org:8080/uclid/ech_specs.html.
How
Children Benefited From the Early Childhood Initiative (ECI)
Developmental Progress
• On entering the program, 86% of the children were
classified as “high risk” for shortcomings in
overall thinking, language, and social and school-readiness
skills. Fourteen percent of the students were deemed to be
both high-risk and developmentally delayed, which would qualify
them for early intervention or special education services
in Pennsylvania. The documented national rate for developmental
delays is 3% to 8%.
• The longer that children participated in high quality
ECI programs, the greater the developmental progress and achievement
of early school success skills.
• After nearly three years in the program, the high-risk
group showed at least average developmental progress without
the typical setbacks for children of poverty documented in
national research.
• The delayed group showed an accelerated rate of developmental
progress into the average range that was 160% of the typical
or expected rate in normal child development.
Social
and Behavioral Progress
• ECI children in the full high-risk group achieved
normal social skills and self-control behaviors compared to
national peers.
• 18% of the children at entry into ECI showed significant
problems with social skills and self-control behaviors that
would qualify them for mental health diagnosis and support;
this challenging behavior problem-group achieved normal social
and behavioral skills after nearly three years of ECI participation.
Early
School Success
• 125 of the children in the ECI program transitioned
to kindergarten and first grade over this period.
• In the school districts from which students were recruited,
an average of 23% of children are retained or “held
back” in kindergarten and first grade, and 21% are referred
to special education programs. After nearly three years of
ECI participation, less than 2% were retained and less than
1% were referred for special education.
• End-of-year “blind” follow-up assessments
by kindergarten and first grade teachers on the Basic School
Skills Inventory-Revised, a nationally standardized achievement
test of early learning skills, demonstrated that ECI children
who transitioned to school performed at an average to above-average
range compared to their national peers.
1
Yoshikawa, H., Rosman, E. A., & Hsueh, J. (2002). Resolving
paradoxical criteria for the expansion and replication of
early childhood care and education programs. Early Childhood
Research Quarterly, 17(3), 3–27.
2 Ramey, C. T., & Ramey, S. L. (1998). Early intervention
and early experience. American Psychologist, 53(2), 109–120.
3 Bagnato, S. J., Suen, H. K., Brickley, D., Smith-Jones,
J., & Dettore, E. (2002). Child developmental impact of
Pittsburgh’s Early Childhood Initiative (ECI) in high-risk
communities: First-phase authentic evaluation research. Early
Childhood Research Quarterly, 17(4), 559–580.
4 HLM (Hierarchical Linear Modeling) is an analysis that estimates
the effects of social units—groups, neighborhoods, schools,
workplaces, organizations, communities, social networks, or
whole social systems—on individuals. Path analysis refers
to the method by which the path of the cause and effect relationship
among variables is determined.
Stephen J. Bagnato, Ed.D.
Professor of Pediatrics & Psychology
Director, Early Childhood Partnerships
Children’s Hospital of Pittsburgh
The UCLID Center at the University of Pittsburgh
Tel: 412-692-6520
Email: steve.bagnato@chp.edu
Robert
Grom
President and CEO
Heritage Health Foundation, Inc.
Greater Braddock Early Childhood Network
Leon
Haynes
President and CEO
Hosanna House
Wilkinsburg Early Childhood Initiative |
