Instructional Sensitivity of Test Items in Educational Testing

• Applicant: Professor Dr. Johannes Hartig
• Subject Area: General and Domain-Specific Teaching and Learning
• Term: from 2015 to 2019
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 262600535

Students' performance in educational tests is commonly attributed to more or less effective teaching. This implies that standardized tests used as outcome measures are instructionally sensitive, i.e. responses to test items are actually affected by instruction the students have received. However, this assumption is rarely investigated empirically. To investigate this assumption empirically, different operationalizations of instructional sensitivity have been proposed. For the proposed project we are building on previous work by Naumann, Hochweber, and Hartig (2013), who proposed a longitudinal multilevel-DIF model to combine two existing yet independent approaches to evaluate items' instructional sensitivity. The model incorporates of changes in performance over time and differences between classrooms, allowing the distinction of global and differential sensitivity. The proposed project aims at four major objectives, the first two of them being primarily of methodological nature: (1) the further development and extension of the LML-DIF model, and (2) the examination of quality and conditions of parameter estimation. Two further objectives are related to substantive questions of classroom assessment test development: (3) the validation of the statistical indicators for instructional sensitivity, and (4) the development of criteria to classify instructional sensitivity on test and item level. The work packages to achieve these goals can be grouped into model development, simulation studies, secondary analyses of existing data sets, and an empirical study conducted at Swiss schools. With respect to model development, we aim at extending the range of accessible research questions by a) a more explicit incorporation of instructional sensitivity on the test level, b) the incorporation of predictor variables, c) the incorporation of varying item discriminations, d) the generalization to polytomous responses, e) and the application to trend data. Simulation studies will support the model development and aim at gaining a deeper knowledge of the quality and conditions (e.g. required sample sizes) of model parameter estimation. Secondary analyses of existing test data will serve to provide criteria for "high" and "low" instructional sensitivity by applying instructional sensitivity models to as many suitable data sets as possible. Furthermore, we aim to validate the statistical indicators for instructional sensitivity by examining their relations with item and classroom level predictors. Additional of secondary analyses are planned to test newly developed instructional sensitivity models in empirical applications. Since available datasets lack several variables desirable for our research questions, we are planning to collect new test data in an empirical study in the Canton St. Gallen, Switzerland. The focus of this study will be to detailed measures of instructional content and exhaustive ratings of test items regarding their relation to instruction.

• DFG Programme: Research Grants
• International Connection: Switzerland
• Partner Organisation: Schweizerischer Nationalfonds (SNF)
• Participating Person: Professor Dr. Jan Hochweber