Construction of Instructionally Sensitive Test Items

• Applicant: Professor Dr. Johannes Hartig
• Subject Area: General and Domain-Specific Teaching and Learning
• Term: from 2018 to 2023
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 413256626

Over the last decades, assessments of students’ achievement and competencies have found widespread use in educational research and policy. For example, policy-makers rely on student test scores to hold schools accountable for their action (e.g., Every Student Succeeds Act of 2015). Educational researchers regularly use student test scores as dependent variables to identify individual, classroom- or school-level characteristics that foster or impair students’ learning. Moreover, students’ achievement is the most commonly applied criterion for judging the effectiveness of specific teaching quality aspects. That is, both educational researchers and policy-makers expect tests to provide beneficial information on students’ individual abilities or the effectiveness of teaching (Pellegrino, 2002).This interpretation requires that instruments are capable of capturing effects of classroom instruction, that is, that tests and items are instructionally sensitive (Popham, 2007). Thus, instructional sensitivity is essential for the validity of the use of test scores as criteria for teaching effectiveness. Yet despite the availability of sound and coherent ways of measuring instructional sensitivity of existing tests and test items, there is little knowledge on how to construct new instructionally sensitive items. Thus, the proposed project aims at two main objectives, (1) the investigation of item properties that enhance instructional sensitivity and (2) a closer examination of instructional variables that link classroom instruction and responses in standardized test items. The content area to be studied in the project is linear equations as taught in mathematics instruction at secondary level. Five project milestones are defined, each marking the successful completion of successive phases of the project. These main phases are (I) the identification and specification of item properties by means of expert interviews and secondary analyses, (II) the development of test items systematically varying with respect to the identified item properties, (III) the collection of student achievement data as well as classroom artefacts, (IV) analyses to answer the main research questions, and (V) the publication of results for audiences both from educational measurement as well as from instructional research.Being able to purposefully build instructionally sensitive items will help to ensure the instructional sensitivity of tests without extensive pretesting or relying on existing instruments, thus being more economically beneficial. Moreover, knowledge on how to construct instructionally sensitive items will allow adequately substantiating validity arguments of educational assessments.

• DFG Programme: Research Grants
• International Connection: Switzerland
• Cooperation Partner: Professor Dr. Jan Hochweber
• Co-Investigator: Dr. Alexander Naumann