Assessing Students' Higher-Order Thinking Skills
Assessing Students' Higher-Order Thinking Skills through Mechanistic Case Diagrams
Kristi Ferguson, PhD Fred Dee, MD Clarence Kreiter, PhD Thomas Haugen, MD, PhD University of Iowa Carver College of Medicine
Funding
Project funded by the Edward J. Stemmler Medical Education Fund of the National Board of Medical Examiners July 1, 2016 through June 30, 2018 Authors have no conflicts of interest to declare
Background
Medical school curricula are moving from discipline-based courses to more integrated approaches Need assessment tools that match this shift How students use basic science content in clinical reasoning would provide valuable evidence for the effectiveness of a specific integration strategy (Kulasegaram and others, 2013)
Purpose of project
to determine whether a computer-based mechanistic case diagramming technique can reliably assess whether students can demonstrate an integrated understanding of concepts learned
Methods
Content expert creates case
Expert identifies and puts in boxes
described in a clinical summary cases with significant underlying pathophysiology are ideal clinical findings and predisposing factors etiologic, pathogenic and pathophysiologic factors that cause the disease and lead to the clinical findings Places the boxes in a logical sequence, drawing directional arrows that connect the boxes in a flow diagram
Each arrow represent relationships such as “causes”, “leads to”, “facilitates”, “predisposes to” or “results in” Students then re-create this diagram on computers by dragging the boxes onto the screen and connecting them with arrows.
Faculty create a list of risk factors, clinical findings, test and procedure findings and therapy (yellow) found in the summary. Then add pathogenic, pathologic, and pathophysiologic mechanisms to the list. Then draw a consensus key
Opening screen for student after reading the clinical summary
Student creates a diagram and finalize. Then they may get feedback.
Student feedback: blue are student arrows that match faculty consensus; red don’t match, and green are faculty consensus arrows not drawn by student. At right are # of matching arrows/total possible, and a narrative description.
Sample
Diagrams were used during Case Analysis in an integrated curriculum N=359 students over two academic years
Scoring methods
students receive one point for every correctly-connected pair of items overall percent correct=the number of correctly-connected pairs identified by the student / total number of pairs identified by the content expert
Analysis
item statistics to assess the reliability and validity of student scores generalizability analysis to assess the stability of overall scores
RESULTS Difficulty and discrimination for 16 cases Difficulty (% correct)
Celiac disease Barrett’s esophagus
73 72
Discrimination (correlation of individual case score with overall score) .50 .33
Parathyroid, hyperparathyroidism
72
.47
Graves disease
71
.38
Lymphoma
69
.33
Peptic ulcer
64
.40
Ectopic pregnancy
64
.49
Cervical cancer
61
.27
MI
57
.37
Emphysema
56
.51
Pulmonary hyaline membrane disease
55
.42
Neural tube
53
.27
Pneumonia
52
.41
Pancreatitis
50
.43
Ulcerative colitis and cancer
50
.23
Lung granuloma
46
.38
Cronbach’s alpha for overall score=.79
W92 NSAID therapy 98% .30 W90 inhibition of COX-1 94% .32 W91 decr in protective prostaglandins
Case 12
70% .49 W88 decr mucus and alkalinity 56% .29 72% .50 W87 acid and pepsin
56% .29
W89 mucosal cell injury
23% .24
W81 epigastric pain
67% .00
53% .59 W84 ulceration 44% .45
36% .32
W85 submucosa, muscularis and vessel injury 67% .49
66% .30
W704 anemia
W86 hemorrhage into lumen 79% .47 W83 bacterial breakdown of hemoglobin
61% .50 W82 melena
NSAID therapy 98% .46 inhibition of COX-1 92% .48
Case 12
in protective prostaglandins 62% .59
revised mucus and alkalinity 45% .44 75% .38 acid and pepsin
mucosal cell injury 53% .59
26% .09
30% .24 epigastric pain 84% .33
ulceration and submucosal, muscularis and vessel injury
90% .51 69% .36 hemorrhage into lumen
anemia
85% .65
bacterial breakdown of hemoglobin
68% .63 melena
G Study Results Effect
Degrees of Freedom
VC
%
P
250
0.0037830
10
C
15
0.0088190
24
PC
3750
0.0235716
65
Results of D Study 0.8
G coefficient
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
5
10
Number of diagrams
15
20
Discussion
Doing 16 diagrams over the course of a semester yields a reasonably reliable summary score. Diagrams can be developed and implemented relatively inexpensively, and can assess application of knowledge in ways that are different from either OSCE cases or traditional multiple-choice exams.
Future plans
Each diagram takes clinical students approximately 10 minutes to complete Implementing technique in a variety of higher-stakes venues
Kristi Ferguson, PhD Fred Dee, MD Clarence Kreiter, PhD Thomas Haugen, MD, PhD University of Iowa Carver College of Medicine
Funding
Project funded by the Edward J. Stemmler Medical Education Fund of the National Board of Medical Examiners July 1, 2016 through June 30, 2018 Authors have no conflicts of interest to declare
Background
Medical school curricula are moving from discipline-based courses to more integrated approaches Need assessment tools that match this shift How students use basic science content in clinical reasoning would provide valuable evidence for the effectiveness of a specific integration strategy (Kulasegaram and others, 2013)
Purpose of project
to determine whether a computer-based mechanistic case diagramming technique can reliably assess whether students can demonstrate an integrated understanding of concepts learned
Methods
Content expert creates case
Expert identifies and puts in boxes
described in a clinical summary cases with significant underlying pathophysiology are ideal clinical findings and predisposing factors etiologic, pathogenic and pathophysiologic factors that cause the disease and lead to the clinical findings Places the boxes in a logical sequence, drawing directional arrows that connect the boxes in a flow diagram
Each arrow represent relationships such as “causes”, “leads to”, “facilitates”, “predisposes to” or “results in” Students then re-create this diagram on computers by dragging the boxes onto the screen and connecting them with arrows.
Faculty create a list of risk factors, clinical findings, test and procedure findings and therapy (yellow) found in the summary. Then add pathogenic, pathologic, and pathophysiologic mechanisms to the list. Then draw a consensus key
Opening screen for student after reading the clinical summary
Student creates a diagram and finalize. Then they may get feedback.
Student feedback: blue are student arrows that match faculty consensus; red don’t match, and green are faculty consensus arrows not drawn by student. At right are # of matching arrows/total possible, and a narrative description.
Sample
Diagrams were used during Case Analysis in an integrated curriculum N=359 students over two academic years
Scoring methods
students receive one point for every correctly-connected pair of items overall percent correct=the number of correctly-connected pairs identified by the student / total number of pairs identified by the content expert
Analysis
item statistics to assess the reliability and validity of student scores generalizability analysis to assess the stability of overall scores
RESULTS Difficulty and discrimination for 16 cases Difficulty (% correct)
Celiac disease Barrett’s esophagus
73 72
Discrimination (correlation of individual case score with overall score) .50 .33
Parathyroid, hyperparathyroidism
72
.47
Graves disease
71
.38
Lymphoma
69
.33
Peptic ulcer
64
.40
Ectopic pregnancy
64
.49
Cervical cancer
61
.27
MI
57
.37
Emphysema
56
.51
Pulmonary hyaline membrane disease
55
.42
Neural tube
53
.27
Pneumonia
52
.41
Pancreatitis
50
.43
Ulcerative colitis and cancer
50
.23
Lung granuloma
46
.38
Cronbach’s alpha for overall score=.79
W92 NSAID therapy 98% .30 W90 inhibition of COX-1 94% .32 W91 decr in protective prostaglandins
Case 12
70% .49 W88 decr mucus and alkalinity 56% .29 72% .50 W87 acid and pepsin
56% .29
W89 mucosal cell injury
23% .24
W81 epigastric pain
67% .00
53% .59 W84 ulceration 44% .45
36% .32
W85 submucosa, muscularis and vessel injury 67% .49
66% .30
W704 anemia
W86 hemorrhage into lumen 79% .47 W83 bacterial breakdown of hemoglobin
61% .50 W82 melena
NSAID therapy 98% .46 inhibition of COX-1 92% .48
Case 12
in protective prostaglandins 62% .59
revised mucus and alkalinity 45% .44 75% .38 acid and pepsin
mucosal cell injury 53% .59
26% .09
30% .24 epigastric pain 84% .33
ulceration and submucosal, muscularis and vessel injury
90% .51 69% .36 hemorrhage into lumen
anemia
85% .65
bacterial breakdown of hemoglobin
68% .63 melena
G Study Results Effect
Degrees of Freedom
VC
%
P
250
0.0037830
10
C
15
0.0088190
24
PC
3750
0.0235716
65
Results of D Study 0.8
G coefficient
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0
5
10
Number of diagrams
15
20
Discussion
Doing 16 diagrams over the course of a semester yields a reasonably reliable summary score. Diagrams can be developed and implemented relatively inexpensively, and can assess application of knowledge in ways that are different from either OSCE cases or traditional multiple-choice exams.
Future plans
Each diagram takes clinical students approximately 10 minutes to complete Implementing technique in a variety of higher-stakes venues
