Using Core Competencies for Assessment and Assurance
STUDENTS BECOMING PROFESSIONALS : Using Core Competencies for Assessment and Assurance
April 2016
SESSION OBJECTIVES • Demonstrate commitment to student professional development by modeling authentic assessment • Support teaching and mentoring in a transparent process of formative assessment • Model program effectiveness through core competency outcomes measures GOALS OF OUR PROJECT: § To leverage the Core Competency outcomes as direct measures of program effectiveness § To discern student progression over time by evidence of scientific and professional growth § To assess the congruence of faculty and student assessment of student achievement
FOCUS of the Session: What does assessment have to do with accreditation?
ISSUES to address
A. Every program has expectations that subgroups of students do not achieve. 1 B. Programs lack effective, formative assessment tools to identify missed achievement as it occurs.
INTRODUCING VAIGS: a unique graduate school environment
I. Historical and Scientific Background Van Andel Institute Graduate School http://vaei.vai.org/grad-school/about-vaigs/ A. Young, private, research institute: focused on human disease (cancer, Parkinson’s, bone, cardio) B. Size: small faculty (33 total laboratories); smaller student body (22 total PhD students) C. Translational focus: our discoveries should “turn into” drugs, diagnostics, biomarkers D. Lean administration: one Dean/President; one Associate Dean (Faculty Dev., Curriculum); Specialists (student affairs, records, admissions)
II. Professional expectations @ VAIGS
A. “Doing” science B. Professional and ethical behavior 2 III. Principles for meaningful assessment A. Assess based on a developmental perspective of student learning B. Clearly align what is taught and what is assessed C. Teachers manage and use assessment data D. Assess with standards of validity and reliability
PROGRAMMATIC CHALLENGES: •
How do FACULTY know what students are supposed to be doing? 3, 4
•
How do the STUDENTS know when they have done it?
presented by Van Andel Institute Graduate School @ the HLC Annual Conference Chicago, IL
STEP #1: Define competencies in your program level or degree
Often informal (or “unwritten”) curriculum These expectations separate top students from the remainder Not what students KNOW, but what they DO
• • •
STEP #2: Grade the accomplishment of these competencies A. Rubric options: • Developmental progression • Hierarchy
B. National / Professional expectations
STEP #3: Formative evaluation of the student A. Student and faculty complete core competencies rubric separately. B. Faculty and student meet to discuss agreement as well as discordance.
STEP #4: Deidentified collection and analysis by cohort
EXERCISE: Build YOUR Program’s Core Competencies Competencies
Domain
Skills demonstrated
1:
a. b. c.
2:
a. b. c. a.
3:
b. c.
Outcomes Data
(WHAT is to be shown?) (How do you know when they accomplish?) (What data will result? How can we use them?)
• • •
• • • • • • • • • • • • • • •
• • • • • • • • •
USING THE OUTCOMES: A. Value to your program • Student outcomes / achievement • Program review / institutional assessment
B. How to deliver this message to your institution l Faculty interest in student success l Recruiting to your program
REFERENCES 1 Lovitts, B. (2007) Making the implicit explicit: creating performance expectations for the dissertation. (Sterling, VA: Stylus Press). 2 Wilson, M. (U.C.Berkeley) “BEAR Assessment” (cited 2/7/15) http://bearcenter.berkeley.edu/page/bear-assessment-system 3 NRC. (2001) Knowing what students know. (cited 2/7/15) http://www.nap.edu/openbook.php?isbn=0309072727 4 Walker, G.E., C.M. Golde, L.Jones, A.C.Bueschel, P.Hutchings. (2008) The Formation of Scholars (San Francisco: Jossey-Bass).
CORE COMPETENCIES for more information: Steve Triezenberg, [email protected]
Julie D.Turner, [email protected] Patty Farrell-Cole, [email protected]
VAIGS CORE COMPETENCIES v2
The student can and does: KNOWLEDGE
Exceptional
Heightened
Evaluate new challenges or Recognize and apply key concepts as they appear in new settings
Describe key concepts in controversies relevant to biomedical science concepts
Place core concepts in Use clinical information to elicit gaps in collective the relevant clinical knowledge context Know scientific Evaluate literature beyond literature relevant to the the dissertation research area research area
RESEARCH
Exceptional
Recognize and apply key concepts as they appear in new clinical settings
Recognize alternative and testable hypotheses
Apply creative and appropriate experimental design
Invent and apply alternative strategies and troubleshoots potential outcomes Analyze unexpected results and allow new discoveries to emerge Generate high technical productivity, publicationquality figures; direct others in technique Defend position or hypothesis-driven experiments to expand the field of study Create new models that integrate experimental findings with external data; Critique data and analysis of others
Use controls appropriately Execute experiments with technical skill Demonstrate critical analysis and thinking Integrate results into relevant models COMMUNICATION
© Van Andel Institute Graduate School 2015, version 2
Speak effectively
Write effectively
Communicate to diverse audiences
Exceptional
Heightened
Manage data with scientific integrity
Independently generate testable hypotheses coupled with specific aims
Demonstrate integrity in shared data; mentor others in appropriate data management Lead collaborative writing
Advancing Use rationale to justify priorities for daily work
Address ethical problems in scientific research Comply with safety and regulatory standards in laboratory activities Display appropriate lab citizenship Work collegially and effectively as a team / collaborator
Intermediate Understand rationale provided by others
Beginning Perform experiments as instructed without considering rationale
Generate independent but Contribute to formation of Accept hypothesis from incomplete hypotheses hypotheses others Execute experiments designed by others
Apply rigorous controls and use outcomes to redirect experimental path Master technical understanding and troubleshoot experimental failures independently
Incorporate both positive and negative controls consistently
Compare contrary results, yielding multiple new hypotheses
Observe limits of current models; identify gaps in knowledge
Describe experiments that Accept current models as generated current model accurate and sufficient
Elucidate logical flaws in Revise extant models based on current models based on new data from self or others. experimental data
Compare new data with Accept only those results prior results from self, consistent with same lab, other labs hypothesis. VAIGS Core Competencies – 2015
Use controls for most approaches
Exhibit appropriate Anticipate results and experimental technique obtain reproducible results and seek advice when experiments fail
Perform controls as instructed by others Receive instruction and practice new skills with supervision
Page 2
Heightened
Advancing
Intermediate CoreCompetencies-122915v2.docx Beginning
Present clear, well-organized research and journal club talks or posters with effective use of schematic and data figures/conclusions
Plan and present journal Plan and present short oral club or research-in-progress summaries in lab meetings talks for lab or center using or mentor meetings simple or pre-set figures
Build coherent and logical argument describing or proposing research studies
Employ standard structures of scientific communication (poster, manuscript, grant formats) with accurate attribution of citations
Describe research purpose and outcomes to small audience of lay adults
Convey goal, rationale, Describe experimental approach, and outcomes to purpose and outcomes to scientists beyond lab group lab peers and supervisor
Coordinate multiple formats Rectify issues where data were for data collection and stor- Record data in prescribed not handled or recorded approage (notebook, electronic format in timely, accurate, priately; revise record-keeping files, artifacts, sample and complete manner methods to prevent issues identity) Develop or employ planning
Engage in best authorship team; serve as submitting Negotiate scope and structure of process for authoring author; conduct quality peer project or manuscript with mulpractices projects; complete outline
Beginning
Connect clinical issues and Describe clinical context of Describe thesis research molecular mechanisms molecular mechanisms as and disease association in accurately and broadly presented by others general and limited terms
Seek expert advice & glean Critically evaluate and modify Collect and employ novel approaches from approaches appropriate approaches literature
Present effective poster or platElegant communication in form presentations at confermultiple settings (conferences; handle questions conences, seminars, classroom, cisely and effectively; manage mass media, ad hoc). logistics and disruptions Present complex models or proposals using elegant Present complex models or language in multiple formats rationale in cogent, convincing, (essay, grant, research or and concise language review article) Communicate science (one’s Communicate research own or other) effectively to effectively to larger and diverse varied audiences (scientists, audiences, wider age ranges clinicians, lay)
ETHICAL / PROFESSIONAL PRACTICE
Intermediate
Describe general concepts Describe general concepts correctly and clearly with gaps and errors
Compare contradictory Understand and evaluate a Understand review article Understand biomedical publications in the dissertation primary research article descriptions of emerging textbook descriptions of research area concepts key concepts
Define sound rationale / Effectively defend given Build expanded rationale for rationale against objections identify gap in project and experiments or alternatives knowledge Frame appropriate hypothesis
Advancing Explain concepts with expanded historical and contextual detail
Participate in design, writing, and editing of collaborative proposals or papers
Write sentences, paragraphs, and essays using established structure and English language standards
Record experimental results with flaws in timeliness, accuracy, and organization
Complete small assigned writing tasks on time for review of manuscripts or tiple contributors; work out and initial draft; effectively editing by others proposals; resolve timeline/writing responsibilities edit drafts from others authorship conflicts Counsel others effectively on Apply effective approaches to Identify the various stake Recognize ethical problems Recognize that ethical ways to identify, work address ethical problems; utilize holders in ethical dilemmas; in case studies; describe problems can arise in through, and resolve ethical support structures for best identify key ethical principles processes to discuss, advise, science problems outcome relevant to given situation or resolve ethical issues Write protocols compliant with Complete training for Complete minimal lab Serve on institutional safety Seek counsel from regulatory regulatory standards for animal, animal, human, safety training as required or research review staff to improve studies and human, hazardous, or recombinant DNA, and apply safe practices in committees protocols recombinant DNA research hazardous research laboratory setting Serve effectively in Seek guidance from peers Understand implications of Serve as role model and mentor Treat others with respect; leadership role in lab and mentors with regard to one’s behavior/attitude in in lab skills and professional follow laboratory rules and operations and/ or institute interpersonal interactions lab and other settings; fulfill behavior standard practices initiatives and conflict resolution assigned lab duties Present and accept honest and Take individual initiative in Work with lab members on Design and lead collaborative Work on individual project; professional feedback in settings obtaining input or collabor- shared project with defined projects to successful accept direction and with multiple individuals; share ation from colleagues for individual responsibilities outcome oversight from others primary project responsibility specific projects assigned by others
© Van Andel Institute Graduate School – 2015
April 2016
SESSION OBJECTIVES • Demonstrate commitment to student professional development by modeling authentic assessment • Support teaching and mentoring in a transparent process of formative assessment • Model program effectiveness through core competency outcomes measures GOALS OF OUR PROJECT: § To leverage the Core Competency outcomes as direct measures of program effectiveness § To discern student progression over time by evidence of scientific and professional growth § To assess the congruence of faculty and student assessment of student achievement
FOCUS of the Session: What does assessment have to do with accreditation?
ISSUES to address
A. Every program has expectations that subgroups of students do not achieve. 1 B. Programs lack effective, formative assessment tools to identify missed achievement as it occurs.
INTRODUCING VAIGS: a unique graduate school environment
I. Historical and Scientific Background Van Andel Institute Graduate School http://vaei.vai.org/grad-school/about-vaigs/ A. Young, private, research institute: focused on human disease (cancer, Parkinson’s, bone, cardio) B. Size: small faculty (33 total laboratories); smaller student body (22 total PhD students) C. Translational focus: our discoveries should “turn into” drugs, diagnostics, biomarkers D. Lean administration: one Dean/President; one Associate Dean (Faculty Dev., Curriculum); Specialists (student affairs, records, admissions)
II. Professional expectations @ VAIGS
A. “Doing” science B. Professional and ethical behavior 2 III. Principles for meaningful assessment A. Assess based on a developmental perspective of student learning B. Clearly align what is taught and what is assessed C. Teachers manage and use assessment data D. Assess with standards of validity and reliability
PROGRAMMATIC CHALLENGES: •
How do FACULTY know what students are supposed to be doing? 3, 4
•
How do the STUDENTS know when they have done it?
presented by Van Andel Institute Graduate School @ the HLC Annual Conference Chicago, IL
STEP #1: Define competencies in your program level or degree
Often informal (or “unwritten”) curriculum These expectations separate top students from the remainder Not what students KNOW, but what they DO
• • •
STEP #2: Grade the accomplishment of these competencies A. Rubric options: • Developmental progression • Hierarchy
B. National / Professional expectations
STEP #3: Formative evaluation of the student A. Student and faculty complete core competencies rubric separately. B. Faculty and student meet to discuss agreement as well as discordance.
STEP #4: Deidentified collection and analysis by cohort
EXERCISE: Build YOUR Program’s Core Competencies Competencies
Domain
Skills demonstrated
1:
a. b. c.
2:
a. b. c. a.
3:
b. c.
Outcomes Data
(WHAT is to be shown?) (How do you know when they accomplish?) (What data will result? How can we use them?)
• • •
• • • • • • • • • • • • • • •
• • • • • • • • •
USING THE OUTCOMES: A. Value to your program • Student outcomes / achievement • Program review / institutional assessment
B. How to deliver this message to your institution l Faculty interest in student success l Recruiting to your program
REFERENCES 1 Lovitts, B. (2007) Making the implicit explicit: creating performance expectations for the dissertation. (Sterling, VA: Stylus Press). 2 Wilson, M. (U.C.Berkeley) “BEAR Assessment” (cited 2/7/15) http://bearcenter.berkeley.edu/page/bear-assessment-system 3 NRC. (2001) Knowing what students know. (cited 2/7/15) http://www.nap.edu/openbook.php?isbn=0309072727 4 Walker, G.E., C.M. Golde, L.Jones, A.C.Bueschel, P.Hutchings. (2008) The Formation of Scholars (San Francisco: Jossey-Bass).
CORE COMPETENCIES for more information: Steve Triezenberg, [email protected]
Julie D.Turner, [email protected] Patty Farrell-Cole, [email protected]
VAIGS CORE COMPETENCIES v2
The student can and does: KNOWLEDGE
Exceptional
Heightened
Evaluate new challenges or Recognize and apply key concepts as they appear in new settings
Describe key concepts in controversies relevant to biomedical science concepts
Place core concepts in Use clinical information to elicit gaps in collective the relevant clinical knowledge context Know scientific Evaluate literature beyond literature relevant to the the dissertation research area research area
RESEARCH
Exceptional
Recognize and apply key concepts as they appear in new clinical settings
Recognize alternative and testable hypotheses
Apply creative and appropriate experimental design
Invent and apply alternative strategies and troubleshoots potential outcomes Analyze unexpected results and allow new discoveries to emerge Generate high technical productivity, publicationquality figures; direct others in technique Defend position or hypothesis-driven experiments to expand the field of study Create new models that integrate experimental findings with external data; Critique data and analysis of others
Use controls appropriately Execute experiments with technical skill Demonstrate critical analysis and thinking Integrate results into relevant models COMMUNICATION
© Van Andel Institute Graduate School 2015, version 2
Speak effectively
Write effectively
Communicate to diverse audiences
Exceptional
Heightened
Manage data with scientific integrity
Independently generate testable hypotheses coupled with specific aims
Demonstrate integrity in shared data; mentor others in appropriate data management Lead collaborative writing
Advancing Use rationale to justify priorities for daily work
Address ethical problems in scientific research Comply with safety and regulatory standards in laboratory activities Display appropriate lab citizenship Work collegially and effectively as a team / collaborator
Intermediate Understand rationale provided by others
Beginning Perform experiments as instructed without considering rationale
Generate independent but Contribute to formation of Accept hypothesis from incomplete hypotheses hypotheses others Execute experiments designed by others
Apply rigorous controls and use outcomes to redirect experimental path Master technical understanding and troubleshoot experimental failures independently
Incorporate both positive and negative controls consistently
Compare contrary results, yielding multiple new hypotheses
Observe limits of current models; identify gaps in knowledge
Describe experiments that Accept current models as generated current model accurate and sufficient
Elucidate logical flaws in Revise extant models based on current models based on new data from self or others. experimental data
Compare new data with Accept only those results prior results from self, consistent with same lab, other labs hypothesis. VAIGS Core Competencies – 2015
Use controls for most approaches
Exhibit appropriate Anticipate results and experimental technique obtain reproducible results and seek advice when experiments fail
Perform controls as instructed by others Receive instruction and practice new skills with supervision
Page 2
Heightened
Advancing
Intermediate CoreCompetencies-122915v2.docx Beginning
Present clear, well-organized research and journal club talks or posters with effective use of schematic and data figures/conclusions
Plan and present journal Plan and present short oral club or research-in-progress summaries in lab meetings talks for lab or center using or mentor meetings simple or pre-set figures
Build coherent and logical argument describing or proposing research studies
Employ standard structures of scientific communication (poster, manuscript, grant formats) with accurate attribution of citations
Describe research purpose and outcomes to small audience of lay adults
Convey goal, rationale, Describe experimental approach, and outcomes to purpose and outcomes to scientists beyond lab group lab peers and supervisor
Coordinate multiple formats Rectify issues where data were for data collection and stor- Record data in prescribed not handled or recorded approage (notebook, electronic format in timely, accurate, priately; revise record-keeping files, artifacts, sample and complete manner methods to prevent issues identity) Develop or employ planning
Engage in best authorship team; serve as submitting Negotiate scope and structure of process for authoring author; conduct quality peer project or manuscript with mulpractices projects; complete outline
Beginning
Connect clinical issues and Describe clinical context of Describe thesis research molecular mechanisms molecular mechanisms as and disease association in accurately and broadly presented by others general and limited terms
Seek expert advice & glean Critically evaluate and modify Collect and employ novel approaches from approaches appropriate approaches literature
Present effective poster or platElegant communication in form presentations at confermultiple settings (conferences; handle questions conences, seminars, classroom, cisely and effectively; manage mass media, ad hoc). logistics and disruptions Present complex models or proposals using elegant Present complex models or language in multiple formats rationale in cogent, convincing, (essay, grant, research or and concise language review article) Communicate science (one’s Communicate research own or other) effectively to effectively to larger and diverse varied audiences (scientists, audiences, wider age ranges clinicians, lay)
ETHICAL / PROFESSIONAL PRACTICE
Intermediate
Describe general concepts Describe general concepts correctly and clearly with gaps and errors
Compare contradictory Understand and evaluate a Understand review article Understand biomedical publications in the dissertation primary research article descriptions of emerging textbook descriptions of research area concepts key concepts
Define sound rationale / Effectively defend given Build expanded rationale for rationale against objections identify gap in project and experiments or alternatives knowledge Frame appropriate hypothesis
Advancing Explain concepts with expanded historical and contextual detail
Participate in design, writing, and editing of collaborative proposals or papers
Write sentences, paragraphs, and essays using established structure and English language standards
Record experimental results with flaws in timeliness, accuracy, and organization
Complete small assigned writing tasks on time for review of manuscripts or tiple contributors; work out and initial draft; effectively editing by others proposals; resolve timeline/writing responsibilities edit drafts from others authorship conflicts Counsel others effectively on Apply effective approaches to Identify the various stake Recognize ethical problems Recognize that ethical ways to identify, work address ethical problems; utilize holders in ethical dilemmas; in case studies; describe problems can arise in through, and resolve ethical support structures for best identify key ethical principles processes to discuss, advise, science problems outcome relevant to given situation or resolve ethical issues Write protocols compliant with Complete training for Complete minimal lab Serve on institutional safety Seek counsel from regulatory regulatory standards for animal, animal, human, safety training as required or research review staff to improve studies and human, hazardous, or recombinant DNA, and apply safe practices in committees protocols recombinant DNA research hazardous research laboratory setting Serve effectively in Seek guidance from peers Understand implications of Serve as role model and mentor Treat others with respect; leadership role in lab and mentors with regard to one’s behavior/attitude in in lab skills and professional follow laboratory rules and operations and/ or institute interpersonal interactions lab and other settings; fulfill behavior standard practices initiatives and conflict resolution assigned lab duties Present and accept honest and Take individual initiative in Work with lab members on Design and lead collaborative Work on individual project; professional feedback in settings obtaining input or collabor- shared project with defined projects to successful accept direction and with multiple individuals; share ation from colleagues for individual responsibilities outcome oversight from others primary project responsibility specific projects assigned by others
© Van Andel Institute Graduate School – 2015
