Project Details
Projekt
Explaining Heterogeneity in Direct Replications: Just Methodological Artifacts?
Subject Area
Personality Psychology, Clinical and Medical Psychology, Methodology
Term
since 2021
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 464590891
Low replication rates have plagued psychological science for about five years. Large-scale replication projects consistently revealed low replicability rates even for classical psychological studies. Between-study variation of true effects (referred to as heterogeneity) and systematic biases (e.g., publication bias or p-hacking) have been discussed as possible contributing factors. One of the central objectives of this project is to provide a valid assessment of heterogeneity in direct replications of psychological effects. With this objective we aim to contribute to the “what” aspect of the META-REP programme: We will identify effects and, possibly, research areas that are affected by heterogeneity and, therefore, less replicable. Heterogeneity has to be expected in conceptual replications because such replications allow for variations across studies that are of theoretical relevance. This applies to a far lesser extent to direct replications that aim to control all theoretically specified variables and moderators. An assessment of heterogeneity in direct replications was only recently made possible by large-scale preregistered replication projects, enabling researchers to investigate heterogeneity of meta-analytic projects without the threats of publication bias and p-hacking. First results indicated that heterogeneity is indeed smaller in direct than in conceptual replications, but still occurs with considerable frequency. This led to various attempts to identify possible theory-relevant effect-moderating contextual variables. We argue, however, that heterogeneity in direct replications may also have been caused by several statistical and methodological artifacts, that have not been sufficiently investigated. These artifacts include variance restrictions, measurement error in IVs and DVs and sub-optimal statistical data models. Some of these artifacts can be avoided when estimtes of heterogeneity are based on unstandardized effect sizes. The second objective of our proposed project, therefore, is to provide a comprehensive investigation of heterogeneity due to such artifacts in published large-scale replication projects and to determine how such methodological artifacts can generate heterogeneity under various realistic scenarios through simulation studies. Thus, our project aims to answer how methodological artifacts have contributed to heterogeneity in published psychological studies as well as how they can generate heterogeneity in psychological studies. With this objective, we address the “why” aspect of the META-REP programme.
DFG Programme
Priority Programmes
