Using document review and data abstraction to inform management of ...
Using document review and data abstraction to inform management of a federal research program: Lessons, benefits, and challenges found by CDC’s Prevention Research Centers Program Demia Sundra Wright, MPH (PRC Program, CDC) Co-Authors: Stephanie Kamin, MPH and Nicola Dawkins, PhD, MPH (Macro International) Jo Anne Grunbaum, EdD and Alicia Norris, MPH (PRC Program, CDC) AEA 2007, Baltimore, MD The findings of this presentation are the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention 1
Outline of Presentation National evaluation overview Methods for document review & data abstraction Lessons Learned Summary
National evaluation overview
PRC Program Overview A network of academic researchers, public health agencies, and community members that conducts applied research in disease prevention and control. PRCs ¾ Conduct research in a variety of health topics, with a variety of communities ¾ Use community-based participatory research (CBPR) methods
Prevention Research Centers Funded in FY 2005
University of Washington at Seattle
University of Minnesota
Oregon Health and Sciences University
State University of New York at Albany
University of Illinois at Chicago
University of Rochester University of Michigan
Harvard University Boston University Yale University
University of Iowa University of California at Berkeley University of California at Los Angeles San Diego State University
University of Colorado
University of Arizona
University of New Mexico
Columbia University University of Pittsburgh The Johns Hopkins University West Virginia University
St. Louis University
University of Kentucky
University of Oklahoma
University of North Carolina at Chapel Hill University of South Carolina Emory University
Texas A&M University Tulane University University of Texas Houston Health Science Center
University of Alabama at Birmingham University of South Florida
Morehouse School of Medicine
Evaluation Approach NATIONAL EVALUATION ACTIVITIES
Quantitative Indicators 33 PRCs
Qualitative Studies Qualitative Studies Document Review – 33 PRCs Interviews – Sample of PRCs
Document Review – 33 PRCs Interviews – Sample of PRCs
First priority on national accountability Second priority on program improvement Participatory approach
Methods for document review and data abstraction
Method Selection
Chose document review to: ¾ ¾ ¾
Reduce burden on PRCs Take advantage of written documents and public data Understand data (not) provided from required reports
Uses ¾
Systematically describe baseline variables • •
¾
Assess breadth and commonalities Use with indicator data in the future
Clarify future reporting & grant requirements
Users ¾
PRC Program office, PRCs, decision-makers
Topical Areas PRC Characteristics Study (Macro) ¾ Characteristics of organization & infrastructure ¾ Characteristics of PRC partner communities
PRC Researcher and Community Interaction Study (Macro) ¾ Types and structure of PRC community committees
PRC Core Research Study (CDC) ¾ Research project selection, design and methods ¾ Current stage of implementation ¾ Adoption and sustainability
Documents and Data Sources Documents submitted by PRCs ¾ Examples: Application; annual workplans, budgets, and progress reports; organizational model(s); community committee guidelines
Documents from PRC Project Officers ¾ Examples: Conference call notes; site visit summary letters and presentations
National Data Sets ¾ Examples: U.S. Census 2000; National Center for Education Statistics
Web searches ¾ Examples: PRC websites; CDC PRC Program website; University/school websites
Procedures Data collection ¾ Collect and inventory documents ¾ Store and code documents and data sources •
ATLAS.ti software, hard copy, or internet bookmarks
¾ Determine each PRC’s communities ¾ Use documents to answer evaluation questions •
MS Access or Word
Data validation ¾ Develop PRC-specific draft reports ¾ Report review by project officers ¾ Report review by PRCs
Data aggregation across PRCs
Lessons Learned
What Worked Potentially most straightforward for data interpretation ¾ University-related characteristics ¾ Community descriptions ¾ Research methods & design
Most informative documents ¾ Those capturing most recent glimpse of moving target •
E.g. recent workplans/progress reports, site visit documents, PRC Information System data
¾ Standardized data sources •
Census, BLS, NCES
¾ Documents rich in detail •
Applications (sometimes)
What Didn’t Work as Well Potentially most difficult to interpret on its own ¾ ¾ ¾ ¾
Organizational structure Staffing Community committee structure Current implementation of research
Least informative documents: Those easily outdated or not well conceptualized ¾ Applications ¾ Organizational charts ¾ Logic models & narrative
Important Steps in the Method Organization and documentation of data sources Validation with POs and PRCs ¾ Critical to have:
• Involvement and support from them • Conversations that supplement documents
Advisory group input
Summary
Challenges Inconsistency within & across data sources Defining each grantee’s “community” ¾ No standardized data for some definitions of community
Characterizing data as point in time ¾ When constant change is reality
Characterizing aspects of CBPR through written materials
Benefits Developed systematic description of program characteristics Was a lower burden on grantees Provided lessons learned on what methods to repeat and not repeat Will provide useful data to PRC Program office, PRCs, and decision-makers ¾ For program improvement and accountability ¾ Enhanced with interview and indicator data
www.cdc.gov/prc Demia Sundra Wright, MPH Health Education Specialist, PRC Program, Division of Adult and Community Health, National Center for Chronic Disease Prevention and Health Promotion
770-488-5506 [email protected]
Outline of Presentation National evaluation overview Methods for document review & data abstraction Lessons Learned Summary
National evaluation overview
PRC Program Overview A network of academic researchers, public health agencies, and community members that conducts applied research in disease prevention and control. PRCs ¾ Conduct research in a variety of health topics, with a variety of communities ¾ Use community-based participatory research (CBPR) methods
Prevention Research Centers Funded in FY 2005
University of Washington at Seattle
University of Minnesota
Oregon Health and Sciences University
State University of New York at Albany
University of Illinois at Chicago
University of Rochester University of Michigan
Harvard University Boston University Yale University
University of Iowa University of California at Berkeley University of California at Los Angeles San Diego State University
University of Colorado
University of Arizona
University of New Mexico
Columbia University University of Pittsburgh The Johns Hopkins University West Virginia University
St. Louis University
University of Kentucky
University of Oklahoma
University of North Carolina at Chapel Hill University of South Carolina Emory University
Texas A&M University Tulane University University of Texas Houston Health Science Center
University of Alabama at Birmingham University of South Florida
Morehouse School of Medicine
Evaluation Approach NATIONAL EVALUATION ACTIVITIES
Quantitative Indicators 33 PRCs
Qualitative Studies Qualitative Studies Document Review – 33 PRCs Interviews – Sample of PRCs
Document Review – 33 PRCs Interviews – Sample of PRCs
First priority on national accountability Second priority on program improvement Participatory approach
Methods for document review and data abstraction
Method Selection
Chose document review to: ¾ ¾ ¾
Reduce burden on PRCs Take advantage of written documents and public data Understand data (not) provided from required reports
Uses ¾
Systematically describe baseline variables • •
¾
Assess breadth and commonalities Use with indicator data in the future
Clarify future reporting & grant requirements
Users ¾
PRC Program office, PRCs, decision-makers
Topical Areas PRC Characteristics Study (Macro) ¾ Characteristics of organization & infrastructure ¾ Characteristics of PRC partner communities
PRC Researcher and Community Interaction Study (Macro) ¾ Types and structure of PRC community committees
PRC Core Research Study (CDC) ¾ Research project selection, design and methods ¾ Current stage of implementation ¾ Adoption and sustainability
Documents and Data Sources Documents submitted by PRCs ¾ Examples: Application; annual workplans, budgets, and progress reports; organizational model(s); community committee guidelines
Documents from PRC Project Officers ¾ Examples: Conference call notes; site visit summary letters and presentations
National Data Sets ¾ Examples: U.S. Census 2000; National Center for Education Statistics
Web searches ¾ Examples: PRC websites; CDC PRC Program website; University/school websites
Procedures Data collection ¾ Collect and inventory documents ¾ Store and code documents and data sources •
ATLAS.ti software, hard copy, or internet bookmarks
¾ Determine each PRC’s communities ¾ Use documents to answer evaluation questions •
MS Access or Word
Data validation ¾ Develop PRC-specific draft reports ¾ Report review by project officers ¾ Report review by PRCs
Data aggregation across PRCs
Lessons Learned
What Worked Potentially most straightforward for data interpretation ¾ University-related characteristics ¾ Community descriptions ¾ Research methods & design
Most informative documents ¾ Those capturing most recent glimpse of moving target •
E.g. recent workplans/progress reports, site visit documents, PRC Information System data
¾ Standardized data sources •
Census, BLS, NCES
¾ Documents rich in detail •
Applications (sometimes)
What Didn’t Work as Well Potentially most difficult to interpret on its own ¾ ¾ ¾ ¾
Organizational structure Staffing Community committee structure Current implementation of research
Least informative documents: Those easily outdated or not well conceptualized ¾ Applications ¾ Organizational charts ¾ Logic models & narrative
Important Steps in the Method Organization and documentation of data sources Validation with POs and PRCs ¾ Critical to have:
• Involvement and support from them • Conversations that supplement documents
Advisory group input
Summary
Challenges Inconsistency within & across data sources Defining each grantee’s “community” ¾ No standardized data for some definitions of community
Characterizing data as point in time ¾ When constant change is reality
Characterizing aspects of CBPR through written materials
Benefits Developed systematic description of program characteristics Was a lower burden on grantees Provided lessons learned on what methods to repeat and not repeat Will provide useful data to PRC Program office, PRCs, and decision-makers ¾ For program improvement and accountability ¾ Enhanced with interview and indicator data
www.cdc.gov/prc Demia Sundra Wright, MPH Health Education Specialist, PRC Program, Division of Adult and Community Health, National Center for Chronic Disease Prevention and Health Promotion
770-488-5506 [email protected]
