CEFISES Seminar: Vincent Holst, « The Curious Case of the Declining Disruption’s Disappearance in Science »

Livestream https://youtube.com/live/GMSGyFIKt8I

Series: Digital Approaches

Speaker: Vincent Holst (Data Analytics Lab, Vrije Universiteit Brussel)

Title: « The Curious Case of the Declining Disruption’s Disappearance in Science »

Abstract: The research field called Science of Science aims to detect structural patterns in science itself. A prominent example is the widely recognized study by Park et al. (2023) that reported a decline in the disruptiveness of scientific and technological knowledge over time. Their main finding is based on the computation of CD indices, a measure of disruption in citation networks, across almost 45 million papers and 3.9 million patents. However, our recent preprint (Holst et al., 2024) fundamentally questions these results. In this talk, I will elaborate how we detected a software bug that hid a large number of maximally disruptive outliers, which turns out to be the main driver behind the reported decline of disruption. More precisely, the majority of the hidden outliers corresponds to database entries with zero references and the decline in disruptiveness is merely reflecting a relative decrease in those entries. Moreover, while these papers and patents with supposedly zero references are the hidden drivers of the reported decline, their source documents predominantly do make references, exposing them as pure dataset artefacts. I will also explain why the two main robustness checks in the original manuscript, a regression analysis and Monte Carlo simulations, failed to detect the apparent effect of the hidden outliers and elaborate on the challenges that we faced revolving around closed-source databases used in the study by Park et al. (2023).

– Park, M., Leahey, E., & Funk, R. J. (2023). Papers and patents are becoming less disruptive over time. Nature, 613(7942), 138-144.
– Holst, V., Algaba, A., Tori, F., Wenmackers, S., & Ginis, V. (2024). Dataset artefacts are the hidden drivers of the declining disruptiveness in science. arXiv preprint arXiv:2402.14583.