Project Details
Projekt
Models, Measures, Moderators - Exploring and Explaining Heterogeneity in Psychological Replications
Subject Area
Personality Psychology, Clinical and Medical Psychology, Methodology
Term
since 2021
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 464590891
In response to the replication crisis, researchers have attempted to shine a light on the role of heterogeneity in effect sizes. However, most meta-meta-studies of this heterogeneity fall short in terms of methodology. In particular, they often employ standardized effect sizes or fail to relate the degree of heterogeneity to the magnitude of the effect. Both practices, as we discovered in the first funding phase of META-REP, adversely affect the accuracy and interpretability of heterogeneity estimates. Hence, we suggest evaluating heterogeneity using an alternative metric—the coefficient of variation (CV) of unstandardized effects. By employing this measure, in stark contrast to prior meta-meta-studies, we identified substantial and theoretically highly relevant heterogeneity even within direct replications of psychological effects. Building on these findings, we will pursue three objectives in the second funding phase of META-REP: First, we seek to enhance the measurement of heterogeneity in replications and improve the interpretability of corresponding results. Second, we aim to increase the empirical knowledge about heterogeneity in psychological replications. Finally, our third goal is to explain heterogeneity, tracing it back to identified variables. Specifically, we propose four modules to achieve these goals: Module 1: In simulations, we develop a data-generating model that accurately describes the emergence of heterogeneity in effect sizes. The assessment of heterogeneity will then involve utilizing both standard indices and CV to investigate possible bias in heterogeneity estimates and to compare their relative statistical performance. Module 2: Using empirical data from conceptual replications, we aim to disentangle heterogeneity across several experimental factors. We compare heterogeneity a) across different kinds of effects, b) across different operationalisations of the same treatment, and c) with consideration of possible moderators. Module 3: In Modules 1 and 2 we are limited to comparisons across identical outcome measures. However, meta-analyses typically aggregate several different measures, so we expand the simulation scheme and empirical data-base to explore such variation. Module 4: In the first funding phase of this project, we have developed the framework “MetaPipeX” to streamline meta-analytic inference and used it to amass a repository of standardized open data. Guided by modules 1-3, we enhance the framework to integrate conceptual replications, moderator analyses, and similar elements, incorporating data used in modules 1-3. This project’s outcomes will contribute to our understanding of heterogeneity in psychology. By improving estimability and comprehension of replicability and generalizability, the results will also shed light on their relation to theoretical underspecification. Consequently, these findings will address META-REP's fundamental What, Why, and How questions in the context of psychological research.
DFG Programme
Priority Programmes
International Connection
Netherlands, United Kingdom
Cooperation Partners
Dr. Johannes Hönekopp; Professor Dr. Jelte Wicherts
