Article Critique 2.pages AWS
Article Critique 2 December 4, 2013 Daniel Gonzales The University of Memphis - ICL7030
GONZALES-ARTICLE CRITIQUE 2
2 Abstract
The point of this paper is to critique an article that is related to the subject of education. In the following paper, the article is that is described and critiqued is Item-writing Rules: Collective Wisdom. Teaching and Teacher Education, written by Bruce B. Frey, Stephanie Peterson, Lisa M. Edwards, Jennifer Teramoto Pedrotti, and Vicki Peyton. The paper is divided in to sections that discuss the content-headings in the article, namely Review of Literature, Methods, Results, Discussion, and Implications. The article includes a table of forty rules to increase the validity of teacher-made tests, Table 1, and a copy of Table 1 is included in the section of this paper titled Appendix A. Appendix A can be found after the Reference section. Table 1 is compiled of a list, from top-down of most frequently occurring rules that are included in the review of twenty classroom assessment textbooks.
GONZALES-ARTICLE CRITIQUE 2
3
Article Critique 2 Introduction The article that is critiqued in the following sections is by Frey et. al, and is titled Item-writing Rules: Collective Wisdom. Refer to the abstract and reference page for a complete list of the authors of this article. The article includes a table of forty rules to increase the validity of teacher-made tests, however these rules have not been tested experimentally. The rules are derived from the analysis of twenty classroom assessment textbooks to obtain a consensus list of item-writing rules. This critique paper includes the article’s headings: Review of Literature, Methods, Results, Discussion, and Implications. The article names four validity concerns that are addressed by the rules in the Discussion section, including potentially confusing wording or ambiguous requirements, the problem of guessing, test-taking efficiency, and controlling for testwiseness. The forty rules that are agreed upon within the textbooks that are analyzed through a group consensus are recommended practices for writing paper-and-pencil objectively scored classroom assessments (Frey et. al, 2005). Stiggins (1991) states that it may take more than one-third of a teacher’s professional time to write classroom assessments, yet there are few research-based rules to guide teachers in this activity. The article’s authors mention that in constructing classroom tests that produce reliable and valid scores, the classroom assessments need to rely on advice, opinions, experience, and common sense.
GONZALES-ARTICLE CRITIQUE 2
4
Review of Literature Estimates show that there are 54 teacher-made tests used each school year (Marso & Pigge, 1988). The literature review section further states that teachers prefer making and using their own tests, as opposed to using a test they obtain elsewhere. A concern is noted here that teachers have historically received little or no training or support after certification (Herman & Dorr-Bremme, 1984). Valid rules should be followed for quality teacher made tests, but empirical studies establishing the validity of item-writing rules are in short supply, and often inconclusive (Frey et. al, 2005). Some authors are noted as studying empirical articles and textbook endorsements to categorize guidelines for multiple-choice, matching, and alternative-choice (true-false) items. Furthermore, the literature review states there is greater research emphasis on the importance and value of other types of assessments in the classroom, such as performance-based, authentic, formative, and informal (Frey et. al, 2005). The majority of tests that are teacherconstructed include paper-and-pencil, objectively scored formats (Earl, 2003). The article continues with a goal of establishing a list of rules that are valid for writing these objectively scored items.
Methods
Twenty educational assessment textbooks that were analyzed to obtain the rulesof-thumb for constructing assessments and test formatting were broken down to note that fourteen of the textbooks included material on classroom assessment training, and teacher preparation. Atleast one of the authors of this study reviewed each of the textbooks to
GONZALES-ARTICLE CRITIQUE 2
5
identify these guidelines. The authors came to agreement on whether some rules were conceptually similar, to avoid duplication. Only rules concerning objectively scored paper-and-pencil testing formats were chosen, which provided guidelines for four different item formats, specifically multiple-choice, matching, true-false, and completion/ fill-in-the-blank items. This study defined completion formatting as nonmultiple-choice items, which require supplying a very short, objectively scored answer (Frey et. al, 2005). A list of all rules is compiled and ranked, and is included in Table 1 to identify the relative importance of each rule that is measured by its frequency.
Results The Results section of the article focuses on the rules that are listed as numbers one through forty in Table 1. As mentioned previously, Table 1 is included in this article critique in Appendix A. If only one textbook that was being reviewed included a rule, that rule was not included in the table. Table 1 lists the rules, as well as indicates the item format to which it applies. In addition, the table also indicates which of the rules has received research support.
Discussion
This section of the article begins with forty different item-writing rules identified in this search. The rationale for each rule often falls into multiple categories, and all present an overriding concern for the validity of interpretation of the item responses. In the table, the most frequently occurring rules to follow, #1, is “All of the Above” should not be an answer question, and #2 is “None of the Above” should not be an answer
GONZALES-ARTICLE CRITIQUE 2
6
option. Conversely, the two least occurring rules to follow when constructing classroom assessments for teachers are #39 Test copies should be clear, readable and not handwritten, and #40 Stems should be on the left and answer options on the right. As mentioned in the introduction paragraph of this article critique, other validity concerns in the Discussion section are separated in this article into four categories: potentially confusing wording or ambiguous requirements, guessing, rules addressing test-taking efficiency, and rules designed to control for testwiseness.
Potentially Confusing Wording or Ambiguous Requirements
Some test-taking respondents may or may not understand a question or set of instructions, therefore their responses may differ as result of that difference, as opposed to their different underlying level of knowledge or skill. For this reason, the rules that are relevant and pertaining to this potential problem are Rules # 1, 2, 6, 11, 14, 15, 19, 20, 25, 27, 29, and 37.
Guessing
Some respondents may choose a correct answer on an assessment by chance. The result will not be valid in whether he or she knows the material. For this reason, the authors suggest to encourage multiple answer options, however there should not be too many answer options to serve unreasonably. Answers should function properly, and the following rules apply to guessing: Rules # 3, 17, 21, 34, and 36.
GONZALES-ARTICLE CRITIQUE 2
7
Rules Addressing Test-taking Efficiency
Quite a few rules are designated to make the test-taking process simple, brief, and free from distractions. The rules that are highlighted in this section of the article include: Rule # 4, 13, 18, 22, 23, 28, 33, 38, 39, and 40.
Rules Designed to Control for Testwiseness
Test-wise and test-taking strategies that lead to validity concerns include the ability to recognize patterns in answer options, identify unintentional clues, and use of other skills unrelated to the knowledge level or ability which is the intended target of the assessment. Frey et. al (2005) state that validity concerns require that items be constructed in ways that prevent the use of these strategies. Rules that pertain to this goal include: Rule # 4, 8, 10, 12, 16, 24, 26, 31, and 32.
Implications
Marso & Pigge (1989) note that studies suggest that perceived classroom assessment skill and actual skill are unrelated or even negatively correlated. There is often little resources available for teachers, as well as little training in the area of assessment construction. Even if teachers have gone through high-quality classroom assessment training, there is an absence of consistent guidelines on the best way to write test items (Frey et. al, 2005). The authors state that a theoretical approach may be the most valid path toward a list of item-writing rules for classroom assessments.
GONZALES-ARTICLE CRITIQUE 2
8 Conclusion
This article definitely shows a collective effort of providing item-writing rules for teachers of constructing classroom assessments. Table 1 is composed by the authors, who reviewed 20 textbooks for increasing the validity of assessments made by classroom teachers. The item-writing rules listed in the table are in ascending order of most occurring, frequency, in each of the textbooks. The study recommends teachers to practice these rules when writing paper-and-pencil objectively scored tests. There are four validity concerns noted in the Discussion section, with corresponding rules to follow provided. Classroom teachers can gain further insight from reading this article, due the study’s focus on importance of rules to follow for test construction.
GONZALES-ARTICLE CRITIQUE 2
9 References
Earl, L. M. (2003). Assessment as learning: using classroom assessment to maximize student learning. Thousand Oaks, CA:Sage. Frey, B.B., Petersen, S., Edwards, L. M., Teramoto Pedrotti, J., & Peyton, V. (2005). Item-writing Rules: Collective Wisdom. Teaching and Teacher Education. 21, 357-364. Herman, J. L., Dorr-Bremme, D. W. (1984). Teachers and testing: implications from a national study. Draft. Paper presented at the Annual Meeting of the American Educa- tional Research Association, NewOrleans, LA, ERIC Document Reproduction Service No. 244 987. Marso, R. N., Pigge, F., L. (1988). Ananalysisofteacher- made tests: testing practices, cognitive demands, and item construction errors. Paper presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans, LA, ERIC Document Reproduction Service No. 298 174. Marso, R. N., & Pigge, F. L. (1989). The status of classroom teachers’ test construction proficiencies: assessment by teachers, principals, and supervisors validated by analyses of actual teacher-made tests. Paper presented at the Annual Herman, Meeting of the National Council of Measurement in Education, San Francisco, CA, ERIC Document Repro- duction Service No. 306-283. Stiggins, R. J. (1991). Relevant classroom assessment training for teachers. Educational Measurement: Issues and Practice, 10, 7–12.
GONZALES-ARTICLE CRITIQUE 2
10 Appendix A
Table 1
GONZALES-ARTICLE CRITIQUE 2
2 Abstract
The point of this paper is to critique an article that is related to the subject of education. In the following paper, the article is that is described and critiqued is Item-writing Rules: Collective Wisdom. Teaching and Teacher Education, written by Bruce B. Frey, Stephanie Peterson, Lisa M. Edwards, Jennifer Teramoto Pedrotti, and Vicki Peyton. The paper is divided in to sections that discuss the content-headings in the article, namely Review of Literature, Methods, Results, Discussion, and Implications. The article includes a table of forty rules to increase the validity of teacher-made tests, Table 1, and a copy of Table 1 is included in the section of this paper titled Appendix A. Appendix A can be found after the Reference section. Table 1 is compiled of a list, from top-down of most frequently occurring rules that are included in the review of twenty classroom assessment textbooks.
GONZALES-ARTICLE CRITIQUE 2
3
Article Critique 2 Introduction The article that is critiqued in the following sections is by Frey et. al, and is titled Item-writing Rules: Collective Wisdom. Refer to the abstract and reference page for a complete list of the authors of this article. The article includes a table of forty rules to increase the validity of teacher-made tests, however these rules have not been tested experimentally. The rules are derived from the analysis of twenty classroom assessment textbooks to obtain a consensus list of item-writing rules. This critique paper includes the article’s headings: Review of Literature, Methods, Results, Discussion, and Implications. The article names four validity concerns that are addressed by the rules in the Discussion section, including potentially confusing wording or ambiguous requirements, the problem of guessing, test-taking efficiency, and controlling for testwiseness. The forty rules that are agreed upon within the textbooks that are analyzed through a group consensus are recommended practices for writing paper-and-pencil objectively scored classroom assessments (Frey et. al, 2005). Stiggins (1991) states that it may take more than one-third of a teacher’s professional time to write classroom assessments, yet there are few research-based rules to guide teachers in this activity. The article’s authors mention that in constructing classroom tests that produce reliable and valid scores, the classroom assessments need to rely on advice, opinions, experience, and common sense.
GONZALES-ARTICLE CRITIQUE 2
4
Review of Literature Estimates show that there are 54 teacher-made tests used each school year (Marso & Pigge, 1988). The literature review section further states that teachers prefer making and using their own tests, as opposed to using a test they obtain elsewhere. A concern is noted here that teachers have historically received little or no training or support after certification (Herman & Dorr-Bremme, 1984). Valid rules should be followed for quality teacher made tests, but empirical studies establishing the validity of item-writing rules are in short supply, and often inconclusive (Frey et. al, 2005). Some authors are noted as studying empirical articles and textbook endorsements to categorize guidelines for multiple-choice, matching, and alternative-choice (true-false) items. Furthermore, the literature review states there is greater research emphasis on the importance and value of other types of assessments in the classroom, such as performance-based, authentic, formative, and informal (Frey et. al, 2005). The majority of tests that are teacherconstructed include paper-and-pencil, objectively scored formats (Earl, 2003). The article continues with a goal of establishing a list of rules that are valid for writing these objectively scored items.
Methods
Twenty educational assessment textbooks that were analyzed to obtain the rulesof-thumb for constructing assessments and test formatting were broken down to note that fourteen of the textbooks included material on classroom assessment training, and teacher preparation. Atleast one of the authors of this study reviewed each of the textbooks to
GONZALES-ARTICLE CRITIQUE 2
5
identify these guidelines. The authors came to agreement on whether some rules were conceptually similar, to avoid duplication. Only rules concerning objectively scored paper-and-pencil testing formats were chosen, which provided guidelines for four different item formats, specifically multiple-choice, matching, true-false, and completion/ fill-in-the-blank items. This study defined completion formatting as nonmultiple-choice items, which require supplying a very short, objectively scored answer (Frey et. al, 2005). A list of all rules is compiled and ranked, and is included in Table 1 to identify the relative importance of each rule that is measured by its frequency.
Results The Results section of the article focuses on the rules that are listed as numbers one through forty in Table 1. As mentioned previously, Table 1 is included in this article critique in Appendix A. If only one textbook that was being reviewed included a rule, that rule was not included in the table. Table 1 lists the rules, as well as indicates the item format to which it applies. In addition, the table also indicates which of the rules has received research support.
Discussion
This section of the article begins with forty different item-writing rules identified in this search. The rationale for each rule often falls into multiple categories, and all present an overriding concern for the validity of interpretation of the item responses. In the table, the most frequently occurring rules to follow, #1, is “All of the Above” should not be an answer question, and #2 is “None of the Above” should not be an answer
GONZALES-ARTICLE CRITIQUE 2
6
option. Conversely, the two least occurring rules to follow when constructing classroom assessments for teachers are #39 Test copies should be clear, readable and not handwritten, and #40 Stems should be on the left and answer options on the right. As mentioned in the introduction paragraph of this article critique, other validity concerns in the Discussion section are separated in this article into four categories: potentially confusing wording or ambiguous requirements, guessing, rules addressing test-taking efficiency, and rules designed to control for testwiseness.
Potentially Confusing Wording or Ambiguous Requirements
Some test-taking respondents may or may not understand a question or set of instructions, therefore their responses may differ as result of that difference, as opposed to their different underlying level of knowledge or skill. For this reason, the rules that are relevant and pertaining to this potential problem are Rules # 1, 2, 6, 11, 14, 15, 19, 20, 25, 27, 29, and 37.
Guessing
Some respondents may choose a correct answer on an assessment by chance. The result will not be valid in whether he or she knows the material. For this reason, the authors suggest to encourage multiple answer options, however there should not be too many answer options to serve unreasonably. Answers should function properly, and the following rules apply to guessing: Rules # 3, 17, 21, 34, and 36.
GONZALES-ARTICLE CRITIQUE 2
7
Rules Addressing Test-taking Efficiency
Quite a few rules are designated to make the test-taking process simple, brief, and free from distractions. The rules that are highlighted in this section of the article include: Rule # 4, 13, 18, 22, 23, 28, 33, 38, 39, and 40.
Rules Designed to Control for Testwiseness
Test-wise and test-taking strategies that lead to validity concerns include the ability to recognize patterns in answer options, identify unintentional clues, and use of other skills unrelated to the knowledge level or ability which is the intended target of the assessment. Frey et. al (2005) state that validity concerns require that items be constructed in ways that prevent the use of these strategies. Rules that pertain to this goal include: Rule # 4, 8, 10, 12, 16, 24, 26, 31, and 32.
Implications
Marso & Pigge (1989) note that studies suggest that perceived classroom assessment skill and actual skill are unrelated or even negatively correlated. There is often little resources available for teachers, as well as little training in the area of assessment construction. Even if teachers have gone through high-quality classroom assessment training, there is an absence of consistent guidelines on the best way to write test items (Frey et. al, 2005). The authors state that a theoretical approach may be the most valid path toward a list of item-writing rules for classroom assessments.
GONZALES-ARTICLE CRITIQUE 2
8 Conclusion
This article definitely shows a collective effort of providing item-writing rules for teachers of constructing classroom assessments. Table 1 is composed by the authors, who reviewed 20 textbooks for increasing the validity of assessments made by classroom teachers. The item-writing rules listed in the table are in ascending order of most occurring, frequency, in each of the textbooks. The study recommends teachers to practice these rules when writing paper-and-pencil objectively scored tests. There are four validity concerns noted in the Discussion section, with corresponding rules to follow provided. Classroom teachers can gain further insight from reading this article, due the study’s focus on importance of rules to follow for test construction.
GONZALES-ARTICLE CRITIQUE 2
9 References
Earl, L. M. (2003). Assessment as learning: using classroom assessment to maximize student learning. Thousand Oaks, CA:Sage. Frey, B.B., Petersen, S., Edwards, L. M., Teramoto Pedrotti, J., & Peyton, V. (2005). Item-writing Rules: Collective Wisdom. Teaching and Teacher Education. 21, 357-364. Herman, J. L., Dorr-Bremme, D. W. (1984). Teachers and testing: implications from a national study. Draft. Paper presented at the Annual Meeting of the American Educa- tional Research Association, NewOrleans, LA, ERIC Document Reproduction Service No. 244 987. Marso, R. N., Pigge, F., L. (1988). Ananalysisofteacher- made tests: testing practices, cognitive demands, and item construction errors. Paper presented at the Annual Meeting of the National Council on Measurement in Education, New Orleans, LA, ERIC Document Reproduction Service No. 298 174. Marso, R. N., & Pigge, F. L. (1989). The status of classroom teachers’ test construction proficiencies: assessment by teachers, principals, and supervisors validated by analyses of actual teacher-made tests. Paper presented at the Annual Herman, Meeting of the National Council of Measurement in Education, San Francisco, CA, ERIC Document Repro- duction Service No. 306-283. Stiggins, R. J. (1991). Relevant classroom assessment training for teachers. Educational Measurement: Issues and Practice, 10, 7–12.
GONZALES-ARTICLE CRITIQUE 2
10 Appendix A
Table 1
