Key findings of the rebuttal

• What the errors were: A bug in the visualization software used by the original authors caused papers with the maximum disruption score to be omitted from published histograms without warning . More critically, the largest source of error was that older database records in Web of Science frequently had incomplete reference metadata — papers with zero references listed were wrongly classified as "maximally disruptive" by the CD index method . As database quality improved over time, these false maximum scores became rarer, creating the illusion of a decline in disruptiveness .
• Magnitude of the artifact: After filtering out these database errors, the decline largely disappeared. In the largest dataset examined, "the artifacts account for 93% of the reported decline between 1945 and 2010" .

What this reveals about the original study's methodology

• The original result was not robust to basic data-quality checks. The finding was driven overwhelmingly by a database artifact — incomplete reference records in older years — rather than a real shift in science .
• The published figures were misleading due to a software bug that hid the very data points that, when examined, proved to be artifacts .
• The original authors' reply (also published by Nature on the same day) disputes this, stating that their own reanalysis using the critics' own preferred exclusion criteria still shows a statistically and practically significant decline . The authors of the rebuttal, however, note that when they restrict to papers with at least 10 references, they find a slight increase in disruption and conclude that the measure simply cannot support conclusive statements about trends .

The 32-month publication delay

• The Belgian researchers submitted their critique to Nature in October 2023, but the journal took 32 months to accept it for publication .
• During that delay, Nature continued to publish editorials and news features drawing on the original paper's conclusion, and the original study was cited over 1,000 additional times and used in at least 16 policy documents .
• Lead researcher Vincent Ginis noted: "Whereas the software bug was fixed by the open-source community in months, it took years for the scientific correction to be published. And all this time, the journal's policies prohibited us from speaking to the press while our critique was under submission" .

Implications for science policy

• The original paper generated over 300 news stories worldwide and was cited in at least 16 policy documents, influencing narratives about an "innovation slowdown" .
• The rebuttal authors state bluntly that "the methodology and conclusions from 2023 do not provide a sound basis for science policy" .
• The case highlights structural weaknesses in the self-correction process: a software bug was fixed quickly by open-source contributors, yet the formal scientific correction took nearly three years; during that time, a flawed finding continued to shape public discourse and policy .
• It also underscores the danger of drawing sweeping conclusions about scientific progress from bibliometric indicators without rigorous validation against database-quality artifacts.

Key findings of the rebuttal

• What the errors were: A bug in the visualization software used by the original authors caused papers with the maximum disruption score to be omitted from published histograms without warning . More critically, the largest source of error was that older database records in Web of Science frequently had incomplete reference metadata — papers with zero references listed were wrongly classified as "maximally disruptive" by the CD index method . As database quality improved over time, these false maximum scores became rarer, creating the illusion of a decline in disruptiveness .
• Magnitude of the artifact: After filtering out these database errors, the decline largely disappeared. In the largest dataset examined, "the artifacts account for 93% of the reported decline between 1945 and 2010" .

What this reveals about the original study's methodology

• The original result was not robust to basic data-quality checks. The finding was driven overwhelmingly by a database artifact — incomplete reference records in older years — rather than a real shift in science .
• The published figures were misleading due to a software bug that hid the very data points that, when examined, proved to be artifacts .
• The original authors' reply (also published by Nature on the same day) disputes this, stating that their own reanalysis using the critics' own preferred exclusion criteria still shows a statistically and practically significant decline . The authors of the rebuttal, however, note that when they restrict to papers with at least 10 references, they find a slight increase in disruption and conclude that the measure simply cannot support conclusive statements about trends .

The 32-month publication delay

• The Belgian researchers submitted their critique to Nature in October 2023, but the journal took 32 months to accept it for publication .
• During that delay, Nature continued to publish editorials and news features drawing on the original paper's conclusion, and the original study was cited over 1,000 additional times and used in at least 16 policy documents .
• Lead researcher Vincent Ginis noted: "Whereas the software bug was fixed by the open-source community in months, it took years for the scientific correction to be published. And all this time, the journal's policies prohibited us from speaking to the press while our critique was under submission" .

Implications for science policy

• The original paper generated over 300 news stories worldwide and was cited in at least 16 policy documents, influencing narratives about an "innovation slowdown" .
• The rebuttal authors state bluntly that "the methodology and conclusions from 2023 do not provide a sound basis for science policy" .
• The case highlights structural weaknesses in the self-correction process: a software bug was fixed quickly by open-source contributors, yet the formal scientific correction took nearly three years; during that time, a flawed finding continued to shape public discourse and policy .
• It also underscores the danger of drawing sweeping conclusions about scientific progress from bibliometric indicators without rigorous validation against database-quality artifacts.