Opening the File Drawer: Assessing and Understanding Publication Bias in the Economic, Behavioral and Social Sciences by Utilizing two German Academic Access Panels

• Applicants: Dr. Jessica Daikeler; Professor Dr. David Richter
• Subject Area: Empirical Social Research
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 512014619

Publication bias can be regarded as the publication (or non-publication) of study results due to the direction or strength. That is when, for example, statistically significant results are more likely to be published than statistically non-significant results. Consequently, publication bias hinders the assessment of the true state of knowledge on a particular research question, resulting in scientific debates and political decisions that are based on false or inadequate scientific evidence. Several factors impact the emergence of publication bias, most importantly the actors directly involved: authors, editors, and reviewers. In addition, the disciplinary and cultural background affects the likelihood and extent of publication bias as well. In our project, we plan to focus on the various steps of researchers’ decision-making processes, which may lead to the publication of selected results while leaving other results in the “file drawer.” The project has two main objectives: (I) First, it aims to measure the extent of publication bias for the economic, behavioral, and social sciences in Germany. This will be accomplished by contrasting published and unpublished GESIS Panel and SOEP-Innovation Sample (SOEP-IS) study submissions. (II) Second, building upon existing theoretical frameworks on the causes of publication bias, we will empirically test the derived hypotheses and (if necessary) adjust and/or extent the current theoretical models. To investigate researchers’ decision processes, this project will be in the unique position to open the file drawer and observe the entire research cycle. It aims to look closely at research practices during the way from a study proposal to the publication stage by using data from study submissions of external researchers to the GESIS Panel and the SOEP-IS. In these two German access panel infrastructures, external researchers from all disciplines can submit their own studies. However, they must include a thorough description of the theoretical background, the hypotheses, the research design, which are then subject to a peer review. Most successful study submissions result in a corresponding publication; some, however, do not. In this context, 184 successful study submissions from both panels combined allow us to investigate which study characteristics (e.g., seniority and number of authors, experimental vs. non-experimental research design) are associated with a higher likelihood of a successful publication. In addition, we will conduct an author survey to collect further actor-based data on motives, attitudes, and values with respect to the research process. Furthermore, for a randomly selected subset of successful study submissions without any corresponding publication, we will conduct own statistical analyses using GESIS Panel and SOEP-IS data to investigate if they have a high likelihood of producing statistically non-significant results, which might explain the lack of a corresponding publication.

• DFG Programme: Research Grants
• International Connection: USA
• Co-Investigators: Privatdozent Dr. Henning Silber; Dr. Bernd Weiß
• Cooperation Partner: Professor James Druckman, Ph.D.