Project Details
Projekt
Fostering Proactive Replicability in Computational Communication Science via Frontloading Effort and Automating Protocols
Applicants
Dr. Johannes Breuer; Professor Dr. Mario Haim
Subject Area
Communication Sciences
Term
since 2021
Project identifier
Deutsche Forschungsgemeinschaft (DFG) - Project number 464291459
This project will develop, implement, and evaluate innovative solutions for improving the replicability (and reproducibility) of research in the field of Computational Communication Science (CCS). Such solutions are urgently needed as recent meta-scientific research, including our own, has pointed out that communication science in general and CCS in particular suffer from issues that systematically hinder replicability. In our first-phase META-REP project, we found that (a) the distinction between reproduction and replication is not as clear in CCS as in other fields and that (b) there is a substantial mismatch between what materials are provided and what replications require. Both issues are largely due to the studied topics, the methods, and also the data that are commonly used in CCS. That is, the widely applied media content and digital trace data can often not be shared for legal and/or ethical reasons and common data-collection methodology such as for social media data prohibits sharing data altogether due to the involved data providers’ terms of service. In addition, CCS is characterized by a rapid development of its studied topics (e.g., public discourse, news, or platform use). Previous research, including our own, has thus pointed to the important role of the publication and review process for ensuring or improving on replicability and reproducibility. For example, it does make a significant difference if journals or conferences have requirements for the sharing of data and other materials (e.g., code). Building on these key insights from our own META-REP project as well as recent work by others (within META-REP and beyond), the goal of the proposed project for the second funding phase is to develop and test solutions for improving replicability/reproducibility in CCS that also have application potential for other fields. For that, the project will leverage replication/reproduction protocols for fostering “proactive replicability,” in that authors of research publications will be asked to engage in practices that have been shown to improve the replicability/reproducibility of their work already when submitting it for publication. We will create such protocols geared to CCS and develop online tools to semi-automatically complete these protocols using a combination of data mining and large language models (LLMs) to extract necessary information from the authors’ manuscripts and compile suggestions on how to improve on a study’s reproducibility/replicability. The protocol and tools created within the project will be systematically evaluated via two experimental studies in collaboration with a key journal in the field and the respective division of our main international association to assess effects of our solutions on research replicability/reproducibility in CCS. Finally, results will be compiled into guidelines and training materials for journals and editors in CCS and beyond.
DFG Programme
Priority Programmes
