EVALUATION OF MCHB’S COMBATING AUTISM ACT INITIATIVE (CAAI) ACTIVITIES Overview of the Evaluation Design Plan
2
3
Presentation Overview I.
II. III. IV. V. VI.
VII.
Insight Team CAAI Grantees CAAI Objectives and Long-Term Goal Evaluation Approach Data Collection Timeline Q&A Contact Information
4
Insight Team PROJECT ROLE
KEY PERSONNEL
Executive Project Director
Claire Wilson
Project Director
Brittany McGill
Project Manager
Alex Suchman
DBP Evaluation
Brittany McGill
LEND and Training Resource Center Evaluation
Betsy Thorn* Alise Nacson
State Implementation Programs and State Public Health Resource Center
Claire Wilson* Meg Tucker
MCH Autism Research Programs
Anne Peterson *Task Leads
5
About the Evaluator • Insight Policy Research (Insight) is a woman•
• •
•
owned small disadvantaged business 12-year history of conducting research on issues affecting at-risk and vulnerable populations Extensive knowledge of HRSA grant programs Conducted the previous evaluation of CAAI activities We value a collaborative approach to research and evaluation
6
Prior Evaluation Reports Insight produced 5 final reports for MCHB: • 4 program-specific reports documenting each program’s progress
in meeting the goals of the CAAI; draft reports were sent to each grantee for comment • One comprehensive report summarizing the collective contributions of those 4 programs
Insight also prepared HRSA’s contribution to the Interagency Report to Congress: • This report described activities funded under the Combating Autism
Act of 2006 (CAA), and highlighted progress in ASD-related research and services since the enactment of the CAAI • HRSA’s contributions to the next Interagency Report to Congress are due in fall 2013
7
Current CAAI Grantees Training Programs
• 43 LEND programs • 10 DBP programs
Research Networks Program
• 3 Networks: AIR-B, AIR-P, DBPNet • 20 R40 grantees
State Autism Policy Programs National Resource Centers
• 22 State Autism Implementation grants
• National Combating Autism Interdisciplinary Training Resource Center • State Public Health Coordinating Resource Center
8
THE COMBATING AUTISM ACT OF 2006 AND HRSA’S COMBATING AUTISM ACT INITIATIVE
9
Combating Autism Act of 2006 (Public Law 109-416) Appropriated funds to NIH, CDC, and HRSA Funds support ASD research and surveillance; autism education; early detection; and intervention services
CAA also established the Interagency Autism Coordinating Committee Reauthorized in 2011, the current Act sunsets in 2014
10
CAAI Objectives The CAAI includes 5 legislative objectives mandated by the Combating Autism Act of 2006, plus an additional objective that reflects MCHB’s focus on systems of care for CSHCNs.
Legislative Objectives
MCHB Objective
• • • • •
Train professionals Increase awareness Conduct research Develop guidelines and tools Reduce barriers to screening and diagnosis
• Building system capacity to provide coordinated, comprehensive, and community-based care for children and youth with ASD and other DD.
11
CAAI Activities, Outcomes, and Long-Term Goal Activities Training
Outcomes
Long-Term Goal
Early screening
Increasing awareness Conducting research Developing guidelines and tools Building system capacity (3 Cs)
Early, interdisciplinary evaluations
Early, evidencebased interventions for children with a confirmed diagnosis
Enabling individuals with ASD and other DD to reach their full potential
12
EVALUATION OF COMBATING AUTISM ACT INITIATIVE ACTIVITIES
13
Evaluation Approach Describe grantee activities
Measure outputs
Qualitative Data Sources • Progress reports • Semi-structured interviews
Quantitative Data Sources • Surveys
Assess change
• NIRS • Research Network questionnaire • SIG survey
• DGIS items
14
Evaluation Objectives Effectiveness • Describe grantee activities, measure outputs, assess change
Efficiency • Are efforts coordinated across and within grant programs? Are grantees collaborating with other Federal, State, and community-based agencies? Are resources being leveraged to maximize the return on Federal investments?
Sustainability • What changes will become institutionalized? What will be maintained following the end of the grants?
15
Evaluation Approach: Describing Grantee Activities ACTIVITY
PRIMARY PROGRAM PRIORITY
Survey
Progress Reports SSIs DGIS
TRAINING
• •
LEND DBP
√
√
√
√
AWARENESS BUILDING
•
ALL
√
√
√
RESEARCH
•
Autism Intervention Research program
√
√
√
√
GUIDELINE & TOOL DEVELOPMENT
•
Autism Intervention Research program
√
√
BUILDING SYSTEM CAPACITY (3 Cs: coordinated, comprehensive, community-based)
•
SIGs
√
√
√
√
16
Evaluation Approach: Measuring Outputs GRANTEE ACTIVITY
SELECTED QUALITATIVE OUTPUTS
SELECTED QUANTITATIVE OUTPUTS
TRAINING
•
Curriculum enhancements to deepen ASD-related competencies (LEND/DBP)
•
Awareness campaigns (States) Dissemination of findings (Research) Community outreach (LEND/DBP, States)
• • •
Number of materials developed and disseminated Number of events held Number of families and providers reached
AWARENESS BUILDING
• •
•
•
Number of LEND and DBP trainees, by year Number of training events sponsored
RESEARCH
•
Efforts to study effectiveness of interventions for underserved populations
• •
Number of papers published Number of conference presentations
DEVELOP GUIDELINES & TOOLS
• •
Types of guidelines developed Intervention tools tested and validated Target audiences for toolkits
•
Number of guidelines developed and disseminated Number of tools developed and disseminated
Statewide planning structures and partnerships established Regional and community-based service networks developed
•
• IMPROVING SYSTEMS OF CARE
• •
•
Number of medical homes that provide integrated, family-centered care for individuals with ASD
17
Evaluation Approach: Assessing Change Possible indicators of improved access to early screening and diagnosis Potential quantitative indicators: • Number of children that receive interdisciplinary diagnostic evaluation services, by year Potential process indicators: • Model practices that have reduced wait times for ASD services
Possible indicators of improved systems of services Potential quantitative indicators: • Number of children with ASD who have access to a medical home Potential process indicators: • Improved awareness among pediatricians of appropriate referral procedures
Possible indicators of increased use of evidence-based practices • Clinician use of research-based guidelines • Parental use of intervention tools
18
DATA COLLECTION TIMELINE
19
Data Collection Timeline January–June 2013
Insight collects and reviews applications and progress reports to date
June–August 2013
Survey data collection: Training & Research
September 2013
HRSA submits contributions to Interagency report to Congress
June–August 2014
Survey data collection: All grantees Semi-structured interviews: All grantees Collect DGIS data: All grantees
September–December 2014
Data analysis (Sources: Survey data, progress reports, interview data, and DGIS)
January 2015
Draft reports due to MCHB
20
21
Questions?
22
For More Information Claire Wilson
[email protected] Brittany McGill
[email protected]