Handout
The Longitudinal Assessment of Student Learning Outcomes: Meeting the Challenge of Sustained Excellence SACSCOC Summer Institute July 19, 20116
Timothy S. Brophy – Director and Professor, Institutional Assessment University of Florida – Office of the Provost Gainesville, Florida
Today’s Goals To review elements of effective assessment systems in the context of sustained excellence Analyze examples of academic assessment SLOs, measures, results, and use of results over a three year period for undergraduate, graduate, and professional programs
Common Challenges for Sustaining Excellence in Assessment Size and scope • Multiple colleges/departments • Certificate, Undergraduate, Graduate, and Professional programs • Available personnel
Management and Tools
Institutional consistency • Outcomes • Assessment reporting • Cycles
Honoring unit autonomy, disciplinary distinctions, and institutional requirements
Faculty comportment
1
SACSCOC Standard 3.3.1.1 3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional effectiveness) 3.3.1.1 educational programs, to include student learning outcomes
It is important to note that the expectation is for evidence of improvement, not evidence that the planned improvements have been implemented the following year
Part 1: Review of the Elements of an Effective Assessment System
Element 1: Define and Disseminate the Terms Student Learning Outcomes (SLOs) are defined generally as “what students are expected to know and be able to do by completion of their degree program” BASIC ELEMENT 1: define this for your faculty and ensure that this definition is consistent across campus and clearly posted
2
Element 2: Categorical Organizing Framework Undergraduate Content/ Discipline Knowledge
Critical Thinking
Communication
Graduate Content/ Discipline Knowledge Skills
Professional Behavior
Element 3: Recent, Relevant, and Rigorous Student Learning Outcomes reflect the curriculum the discipline, and faculty expectations; as these elements evolve, learning outcomes change. Recent – the outcome reflects current knowledge and practice in the discipline. Relevant – the outcome relates logically and significantly to the discipline and the degree. Rigorous – the degree of academic precision and thoroughness that the outcome requires to be met successfully.
Element 4: Distinguish Outputs from Outcomes Outputs describe and count what we do and whom we reach, and represent products or services we produce. Processes deliver outputs; what is produced at the end of a process is an output.
An outcome is a level of performance or achievement. It may be associated with a process or its output. Outcomes imply measurement – quantification of performance.
3
Element 5: Distinguish SLOs and Program Goals Student Learning Outcomes (SLOs) describe student learning – what students will know and be able to do as a result of completing an academic program (undergraduate, graduate, professional, and certificate). Program faculty set targets for their SLOs
Program Goals describe the unit’s expectations for programmatic elements, such as admission criteria, acceptance and graduation rates, etc. – see p. 4 of your handout
Basic Element 6: Ensure the Outcome is Measurable Effective SLOs:
Focus on what students will know and be able to do. • All disciplines have a body of core knowledge that students must learn to be successful as well as a core set of applications of that knowledge in professional settings. Describe observable and measureable actions or behaviors. • Effective SLOs present a core set of observable, measureable behaviors. Measurement tools vary from exams to complex tasks graded by rubrics. The key to measurability: an active verb that describes an observable behavior, process or product. • A framework for developing SLOs: Bloom’s Taxonomy (pages 10‐13 in your handout)
Verbs and Phrases that Complicate Measurability Understand • An internal process that is indicated by demonstrated behaviors – OK for learning goals but not recommended for program or course SLOs
Appreciate; Value • Internal processes that are indicated by demonstrated behaviors closely tied to personal choice or preference; OK if the appreciation or valuing is supported by discipline‐specific knowledge
Become familiar with • Focuses assessment on “becoming familiar,” not familiarity
4
Verbs and Phrases that Complicate Measurability Learn about, Think about • Not observable; demonstrable through communication or other demonstration of learning
Become aware of, Gain an awareness of • Focuses assessment on becoming and/or gaining – not actual awareness
Demonstrate the ability to • Focuses assessment on ability, not achievement or demonstration of a skill
Element 7: Balance Direct and Indirect Assessments Direct assessments of student learning are those that provide for direct examination or observation of student knowledge or skills against measurable performance indicators.
Indirect assessments are those that ascertain the opinion or self‐report of the extent or value of learning experiences (Rogers, 2011)
Element 8: Assessment Planning Timeline/Cycle What we are reviewing today Develop assessment plans
Implement plan
Submit for institutional review
Collect data and submit reports
Some sample Assessment Plan Review Rubrics used at UF are in your Handout on pp. 14-16.
5
Element 9: Faculty Support ‐ Templates and Rubrics Mission Alignment Student Learning Outcomes
Assessment Oversight
Academic Assessment Plan Curriculum/ Assessment Maps
Methods and Procedures Assessment Cycle
Element 10: Approval and Management Process The University of Florida SLO/Assessment Plan Approval Process
Program/
College
Department
Student Academic Support System
Academic Assessment Committee
University Curriculum Committee
Element 11: Assessment and Data Reporting System Assessment and Institutional Effectiveness Establish Mission, Goals, and Outcomes
Modify and Improve
Interpret and Evaluate the Data
Data Reporting
February – Assessment Plans submitted for the next AY Assessment Planning
October ‐ Assessment Data, results, and use of results for previous AY reported
Implement the Plan and Gather Data
6
Element 12: Quality Assurance Process Multi‐step, institutional review and approval process
Templates and rubrics for guiding faculty through the process
Review and evaluate faculty submissions
Cross‐reference plans with data reported annually
Develop and provide professional development
Model the process: Modify and improve quality assurance processes based on the data you collect
Elements 1. Define the terms and disseminate them 2. Consider an institutional categorical organizing framework for SLOs 3. Recent, Relevant, and Rigorous 4. Distinguish Outputs from Outcomes 5. Distinguish SLOs from Program Goals 6. Ensure the outcome is measurable
Elements 7. Balance direct and indirect assessments 8. Planning Timeline/Cycle 9. Templates and Rubrics 10. Approval and Management Process 11. Develop a System or Cycle of Assessment and Reporting 12. Quality Assurance Process
7
Group activities
The University of Florida Data Reporting Template (p. 8) SLOs: • Cross‐check to ensure these are consistent in the Academic Assessment Plan, the catalog, and the online reporting system • Update SLOs using the appropriate forms in the approval system if needed
Assessment Method: • List the assignment, exam, project, etc. • If this is a sample, describe the sampling procedure used
Results: • Enter the criterion for success, and if the criterion is less than 70%, provide a rationale. • “X number of students passed the assessment out of a total of Y students, for a percentage of Z%” • This meets/does not meet the criterion for success. • Attach the data you shared with your faculty (student names redacted). NOTE: Please have raw data available in case it is requested.
Use of Results: • State who reviewed the results. • Refer to the results that were reviewed. • State actions taken in past tense.
Group Activity 1: Longitudinal SLO analysis BS – Food and Resource Economics Questions: Are the SLOs measurable each year? Results – Is there a criterion for success? Are results aligned with the Assessment Methods? Are Results expressed in terms of the degree to which they meet the criterion for success? Use of results – is there evidence of improvement at least every two years? Based on your analysis, what would you communicate to the faculty?
8
Group Activity 2: Longitudinal SLO analysis DMD – Doctor of Dental Medicine Questions: Are the SLOs measurable each year? Results – Is there a criterion for success? Are results aligned with the Assessment Methods? Are Results expressed in terms of the degree to which they meet the criterion for success? Use of results – is there evidence of improvement at least every two years? Based on your analysis, what would you communicate to the faculty?
Group Activity 3: Longitudinal SLO analysis Architecture ‐ MS Questions: Are the SLOs measurable each year? Results – Is there a criterion for success? Are results aligned with the Assessment Methods? Are Results expressed in terms of the degree to which they meet the criterion for success? Use of results – is there evidence of improvement at least every two years? Based on your analysis, what would you communicate to the faculty?
Group Activity 4: Longitudinal SLO analysis BS – Geography Questions: Are the SLOs measurable each year? Results – Is there a criterion for success? Are results aligned with the Assessment Methods? Are Results expressed in terms of the degree to which they meet the criterion for success? Use of results – is there evidence of improvement at least every two years? Based on your analysis, what would you communicate to the faculty?
9
Questions Timothy S. Brophy, Ph.D. Professor and Director, Institutional Assessment [email protected] 352‐273‐4476 http://assessment.aa.ufl.edu
10
Timothy S. Brophy – Director and Professor, Institutional Assessment University of Florida – Office of the Provost Gainesville, Florida
Today’s Goals To review elements of effective assessment systems in the context of sustained excellence Analyze examples of academic assessment SLOs, measures, results, and use of results over a three year period for undergraduate, graduate, and professional programs
Common Challenges for Sustaining Excellence in Assessment Size and scope • Multiple colleges/departments • Certificate, Undergraduate, Graduate, and Professional programs • Available personnel
Management and Tools
Institutional consistency • Outcomes • Assessment reporting • Cycles
Honoring unit autonomy, disciplinary distinctions, and institutional requirements
Faculty comportment
1
SACSCOC Standard 3.3.1.1 3.3.1 The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results in each of the following areas: (Institutional effectiveness) 3.3.1.1 educational programs, to include student learning outcomes
It is important to note that the expectation is for evidence of improvement, not evidence that the planned improvements have been implemented the following year
Part 1: Review of the Elements of an Effective Assessment System
Element 1: Define and Disseminate the Terms Student Learning Outcomes (SLOs) are defined generally as “what students are expected to know and be able to do by completion of their degree program” BASIC ELEMENT 1: define this for your faculty and ensure that this definition is consistent across campus and clearly posted
2
Element 2: Categorical Organizing Framework Undergraduate Content/ Discipline Knowledge
Critical Thinking
Communication
Graduate Content/ Discipline Knowledge Skills
Professional Behavior
Element 3: Recent, Relevant, and Rigorous Student Learning Outcomes reflect the curriculum the discipline, and faculty expectations; as these elements evolve, learning outcomes change. Recent – the outcome reflects current knowledge and practice in the discipline. Relevant – the outcome relates logically and significantly to the discipline and the degree. Rigorous – the degree of academic precision and thoroughness that the outcome requires to be met successfully.
Element 4: Distinguish Outputs from Outcomes Outputs describe and count what we do and whom we reach, and represent products or services we produce. Processes deliver outputs; what is produced at the end of a process is an output.
An outcome is a level of performance or achievement. It may be associated with a process or its output. Outcomes imply measurement – quantification of performance.
3
Element 5: Distinguish SLOs and Program Goals Student Learning Outcomes (SLOs) describe student learning – what students will know and be able to do as a result of completing an academic program (undergraduate, graduate, professional, and certificate). Program faculty set targets for their SLOs
Program Goals describe the unit’s expectations for programmatic elements, such as admission criteria, acceptance and graduation rates, etc. – see p. 4 of your handout
Basic Element 6: Ensure the Outcome is Measurable Effective SLOs:
Focus on what students will know and be able to do. • All disciplines have a body of core knowledge that students must learn to be successful as well as a core set of applications of that knowledge in professional settings. Describe observable and measureable actions or behaviors. • Effective SLOs present a core set of observable, measureable behaviors. Measurement tools vary from exams to complex tasks graded by rubrics. The key to measurability: an active verb that describes an observable behavior, process or product. • A framework for developing SLOs: Bloom’s Taxonomy (pages 10‐13 in your handout)
Verbs and Phrases that Complicate Measurability Understand • An internal process that is indicated by demonstrated behaviors – OK for learning goals but not recommended for program or course SLOs
Appreciate; Value • Internal processes that are indicated by demonstrated behaviors closely tied to personal choice or preference; OK if the appreciation or valuing is supported by discipline‐specific knowledge
Become familiar with • Focuses assessment on “becoming familiar,” not familiarity
4
Verbs and Phrases that Complicate Measurability Learn about, Think about • Not observable; demonstrable through communication or other demonstration of learning
Become aware of, Gain an awareness of • Focuses assessment on becoming and/or gaining – not actual awareness
Demonstrate the ability to • Focuses assessment on ability, not achievement or demonstration of a skill
Element 7: Balance Direct and Indirect Assessments Direct assessments of student learning are those that provide for direct examination or observation of student knowledge or skills against measurable performance indicators.
Indirect assessments are those that ascertain the opinion or self‐report of the extent or value of learning experiences (Rogers, 2011)
Element 8: Assessment Planning Timeline/Cycle What we are reviewing today Develop assessment plans
Implement plan
Submit for institutional review
Collect data and submit reports
Some sample Assessment Plan Review Rubrics used at UF are in your Handout on pp. 14-16.
5
Element 9: Faculty Support ‐ Templates and Rubrics Mission Alignment Student Learning Outcomes
Assessment Oversight
Academic Assessment Plan Curriculum/ Assessment Maps
Methods and Procedures Assessment Cycle
Element 10: Approval and Management Process The University of Florida SLO/Assessment Plan Approval Process
Program/
College
Department
Student Academic Support System
Academic Assessment Committee
University Curriculum Committee
Element 11: Assessment and Data Reporting System Assessment and Institutional Effectiveness Establish Mission, Goals, and Outcomes
Modify and Improve
Interpret and Evaluate the Data
Data Reporting
February – Assessment Plans submitted for the next AY Assessment Planning
October ‐ Assessment Data, results, and use of results for previous AY reported
Implement the Plan and Gather Data
6
Element 12: Quality Assurance Process Multi‐step, institutional review and approval process
Templates and rubrics for guiding faculty through the process
Review and evaluate faculty submissions
Cross‐reference plans with data reported annually
Develop and provide professional development
Model the process: Modify and improve quality assurance processes based on the data you collect
Elements 1. Define the terms and disseminate them 2. Consider an institutional categorical organizing framework for SLOs 3. Recent, Relevant, and Rigorous 4. Distinguish Outputs from Outcomes 5. Distinguish SLOs from Program Goals 6. Ensure the outcome is measurable
Elements 7. Balance direct and indirect assessments 8. Planning Timeline/Cycle 9. Templates and Rubrics 10. Approval and Management Process 11. Develop a System or Cycle of Assessment and Reporting 12. Quality Assurance Process
7
Group activities
The University of Florida Data Reporting Template (p. 8) SLOs: • Cross‐check to ensure these are consistent in the Academic Assessment Plan, the catalog, and the online reporting system • Update SLOs using the appropriate forms in the approval system if needed
Assessment Method: • List the assignment, exam, project, etc. • If this is a sample, describe the sampling procedure used
Results: • Enter the criterion for success, and if the criterion is less than 70%, provide a rationale. • “X number of students passed the assessment out of a total of Y students, for a percentage of Z%” • This meets/does not meet the criterion for success. • Attach the data you shared with your faculty (student names redacted). NOTE: Please have raw data available in case it is requested.
Use of Results: • State who reviewed the results. • Refer to the results that were reviewed. • State actions taken in past tense.
Group Activity 1: Longitudinal SLO analysis BS – Food and Resource Economics Questions: Are the SLOs measurable each year? Results – Is there a criterion for success? Are results aligned with the Assessment Methods? Are Results expressed in terms of the degree to which they meet the criterion for success? Use of results – is there evidence of improvement at least every two years? Based on your analysis, what would you communicate to the faculty?
8
Group Activity 2: Longitudinal SLO analysis DMD – Doctor of Dental Medicine Questions: Are the SLOs measurable each year? Results – Is there a criterion for success? Are results aligned with the Assessment Methods? Are Results expressed in terms of the degree to which they meet the criterion for success? Use of results – is there evidence of improvement at least every two years? Based on your analysis, what would you communicate to the faculty?
Group Activity 3: Longitudinal SLO analysis Architecture ‐ MS Questions: Are the SLOs measurable each year? Results – Is there a criterion for success? Are results aligned with the Assessment Methods? Are Results expressed in terms of the degree to which they meet the criterion for success? Use of results – is there evidence of improvement at least every two years? Based on your analysis, what would you communicate to the faculty?
Group Activity 4: Longitudinal SLO analysis BS – Geography Questions: Are the SLOs measurable each year? Results – Is there a criterion for success? Are results aligned with the Assessment Methods? Are Results expressed in terms of the degree to which they meet the criterion for success? Use of results – is there evidence of improvement at least every two years? Based on your analysis, what would you communicate to the faculty?
9
Questions Timothy S. Brophy, Ph.D. Professor and Director, Institutional Assessment [email protected] 352‐273‐4476 http://assessment.aa.ufl.edu
10
