Towards neutral comparison studies in methodological biostatistics research

• Applicant: Professorin Dr. Anne-Laure Boulesteix
• Subject Area: Epidemiology and Medical Biometry/Statistics
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 497316619

Statisticians are often keen to analyze the statistical aspects of the so-called “replication crisis”. They condemn fishing expeditions and publication bias across empirical scientific fields applying statistical methods. But what about good practice issues in their own - methodological - research, i.e. research considering statistical methods as research objects? When developing and evaluating new statistical methods and data analysis tools, do statisticians adhere to the good practice principles they promote in fields which apply statistics? I argue that statisticians should make substantial efforts to address what may be called the replication crisis in the context of methodological research in statistics and data science. The first project aim is to increase awareness of issues related to this replication crisis with a focus on biases affecting many methodological studies. This will be achieved in particular through well-designed empirical studies that also help to identify strategies for improvement. In this context, we argue that neutral comparison studies investigating (bio)statistical methods with thoughtfully elaborated design, strategies to avoid various types of bias when evaluating methods and rigorous unbiased reporting play an important role for increasing the reliability of methodological literature and allowing more evidence-based and unbiased choice of statistical/data analysis methods. The second – major - aim of the project is to develop, demonstrate and evaluate potential strategies to design, conduct, interpret and report comparison studies as “neutrally” as possible, whereby the precise definition of the concept of neutrality will also be part of the project. The project is structured in three interrelated subprojects dealing with the neutrality of comparison studies: “design of comparison studies and biases”, “reporting of neutral comparison studies”, and “handling of the flexibility of comparison studies”. Its originality is that it adopts a metascientific approach to address methodological issues in statistics and related fields, where it is unusual to do so. Meta-research is not yet perceived by the methodological scientific community as a full-fledged research area (in contrast, inspiring developments have been observed in other fields such as psychology and health research in the two last decades). The project thus also aims at establishing metascience in the context of methodological research on statistics and data science, which should, in the long run, lead to increased research quality not only in this field but also across all fields that apply statistical/data analysis methods.

• DFG Programme: Research Grants