Experience Paper: "This Paper is Quite Unusual:" Metascience and Structural Misalignment in the Security & Privacy Research Community
Abstract
Metascience examines the practices, evaluation mechanisms, and incentive structures that shape the production and validation of scientific knowledge. While metascientific research is increasingly institutionalized across disciplines, including parts of computer science, its presence in the Security and Privacy (S&P) community remains limited, particularly in flagship venues. This paper analyzes the current state of metascience in S&P and identifies structural factors, notably evaluation criteria that equate novelty with technical innovation, that constrain its integration into the field's core publication ecosystem. We (i) review existing S&P metascientific work and map its publication landscape, (ii) report on our experience submitting metascience studies to major S&P conferences, and (iii) examine the methodological and institutional barriers that arise when a predominantly technical research culture engages in self-reflective research using social-science methods. We argue that prevailing evaluative norms create a structural mismatch with metascientific contributions and propose pathways for integrating metascience into the S&P community.
Keywords
Conferences; Printing; Security; Privacy; Computer security; Reviews; Computers; Publishing; Transparency (climate reporting); Ranking (statistics)