top of page
  • Writer's pictureAlvin Djajadikerta

Diversify the Scientific Publishing Model

Updated: Nov 22, 2023

Moving beyond academic journals could improve the reproducibility, creativity, and accessibility of scientific discoveries.





Introduction


If you’ve spent time in academic science, you’ll have heard plenty of frustration about the publishing process. Although there’s humour to be found in the banality of many papers or struggles with peer review, the grievances of scientists often stem from serious underlying issues. Papers can take months or years to publish, often don’t reproduce, and can be very expensive. Moreover, the pressure to constantly publish creates misaligned incentives that might interfere with some of the best science.


I think these issues can be partly remedied by tackling the homogenous culture of scientific publishing. Prestigious journals now have strikingly similar formats, review processes, and selection criteria. This wasn’t always this way, and it doesn’t have to be moving forward. In the following sections, I argue that diversifying the publishing model by creating different types of publication can help incentivise more accurate and creative science.


How strong is a typical journal article as evidence?


We understand the world by observing our environment and building a corresponding model of reality. But very little of our knowledge today comes from our own direct observations. Instead, we get information through a variety of media: word-of-mouth, television news, online videos, and news articles. These media form a user interface of sorts with reality: we usually can’t access data directly, but must instead do so through the media we choose to consume.


Each form of media has the potential to slightly distort primary observations. This is a lesson most of us learn early on. We know that some types of media tend to be more reliable than others—we know that claims from infomercials, political advertising, and tabloid journalism (for instance) should probably be taken with a pinch of salt, mainly because the motives of their creators might not be aligned with ours.


In science, we almost exclusively learn about new findings through academic journal articles. Although these articles might be held to more rigorous standards than most popular media, they are still best thought of as media, and as such can distort primary observations. The sources of bias in this case are the incentives of researchers and publishers. Researchers are motivated to publish in high-impact journals as a form of career currency, and these journals prefer publishing exciting positive results that will draw in views and citations.


These incentives mean that positive results are much more likely to get published than negatives. In 2007, over 85% of all papers reported positive results, rising from 70% in 1990-91. This bias isn’t because most results tend to be positive—in psychology, for instance, only 44% of pre-registered studies (where bias is less of a factor) were positive, in comparison with 96% of standard reports. This skew towards positive data can lead to the literature becoming biased towards false positives or cargo cult science that appears to validate a popular hypothesis.


This has led to the suggestion that many published research findings could be false—all without any fraud or deception by experimenters necessary. Empirical results seem to confirm this expectation. Replication studies across cancer biology, psychology, the social sciences, economics, and drug discovery find that around ~35-65% of studies produce significant effects in replication studies (often with lower effect sizes). This rate might be even lower for ‘landmark’ studies that describe completely new and exciting phenomena—replication efforts by the pharmaceutical company Amgen were only able to replicate 11% of such studies.


From a reader’s perspective, it may then be wise to exercise some caution when interpreting published findings. Failure to reproduce could be for a number of reasons, and doesn’t necessarily mean a result is “wrong”. However, non-reproducible results can’t be accepted at face value without further examination. The replication percentages we observe are not extremely low (they are above what one might expect by chance), but neither are they high enough to argue that articles generally constitute very strong evidence. In the absence of further information, a journal article thus conveys only a moderate level of certainty, and this is largely determined by the processes by which articles are produced.


There are many situations where more certainty would be valuable. Consumers of scientific research include other academic researchers, pharmaceutical companies, policymakers, doctors, and the general public: many of these groups make important decisions based on science, and would benefit from information they can be more confident in. For example, pharmaceutical companies are often frustrated by the low validation rates of drug targets from academic research, and must spend significant time and money on in-house validation. In academia, over 70% of researchers report not being able to reproduce a published result—again, this represents significant wasted effort, especially as reproduction attempts tend not to be subsequently published. Science does have self-correction mechanisms, but these can be slow, and false results can cause substantial waste in the meantime.


A recipe for higher-certainty publications


How can we be more certain about the accuracy of a result? We could try to reproduce it ourselves, but in practice this is rarely feasible. Even for scientists in the same field, reproducing results is costly and time-consuming; for most others, it’s nearly impossible. What might be more practical, then, is to put more trust in results that have been independently verified by third parties.


In statistical terms, independent replication assures us that data represent truly independent observations. Results produced within one lab might not be completely independent—minor differences in protocol or unconscious bias may lead to spurious results. Journals exacerbate this effect by subsampling the most exciting data to publish, enriching for these spurious results. Replication counters this by directly resampling the underlying distribution of data, ensuring observations are actually independent.


This gives us a path to making higher-certainty publications. If we can be confident that observations aren’t distorted by publication bias, this is almost as good as seeing the results ourselves. Most other methods suggested to improve reproducibility, such as pre-registration of studies, essentially leverage the same principle of ensuring genuine independence.


Achieving this in practice is complex. A number of suggestions have been made to improve reproducibility in publishing. These proposals usually suggest that we should apply more stringent standards (higher statistical significance, pre-registration, more reporting requirements) to scientific publications. While some of these standards might create higher-certainty papers, I think it’s also worth considering their costs. Science has many global goals; among them might be increasing the overall level of human knowledge and developing technologies that improve our collective welfare. Reproducibility is ultimately an instrumental goal on the path to these—a very important one, but instrumental nevertheless.


If increased reproducibility comes at the cost of drastically reducing the total number of scientific findings, the trade-off might not be worthwhile. In exploratory fields, in which the research process is creative, non-linear, and often serendipitous, compliance with overly stringent standards is probably inconsistent with meaningful progress. For some research questions, ethical considerations may also preclude the sample sizes required to meet stringent significance standards. It’s very hard to construct universal rules that would be good for research on the whole.


It’s also uncertain how well such standards would be implemented within the current academic system. The incentives and processes that contributed to the initial problem haven’t disappeared and won’t go away easily. Any new rules would have to be examined carefully to make sure they were relatively resistant to Goodhart’s law—if they are easy to manipulate, they may well make the problem worse.


Diversifying the publishing ecosystem


Instead of any universal solution, I’d suggest that what we need are new classes of publication that aim for different evidentiary standards. We need a diverse ecosystem of publications, aligned along a hierarchy of evidence that makes it easy for us to find the best evidence for different questions.


Hierarchies of evidence are not a radical idea. Some fields have adopted formal criteria for strength of evidence—clinical medicine, for instance, has a hierarchy of evidence that spans from randomised controlled trials to case studies. Although we clearly can’t apply this specific hierarchy to other fields, we can use the principle of statistical independence outlined previously to define a general hierarchy for the experimental sciences. In such a hierarchy, the top tier of evidence would be for independently validated results, pre-registered studies and other formats in which we can be confident that the results aren’t distorted by their production processes.

This hierarchy would create room for science that’s less empirical, or otherwise qualitatively different, than that typically published in journals today. These could include nascent ideas, interesting observations, theoretical articles or books, and negative data—important topics that are often undervalued in science today. A favourite example of mine is the influence of two key theoretical biology books—Schrodinger’s What is Life? and von Neumann’s Theory of Self-Reproducing Automata—on the molecular biology revolution of the 20th century. These works directly inspired the work of Crick, Watson, and Brenner (among others), offering an excellent demonstration of how essential ideas and theories are in biology.


It’s important to note that all levels of this hierarchy are necessary, and none are intrinsically ‘better’ or more important than any other. There are necessary trade-offs between different kinds of research; at any given point in time, the set of questions that can be answered with high certainty is necessarily much smaller than the set we can answer with low certainty. It’s good that people are asking these harder questions, even if we can’t always be absolutely sure about the answers. If more publishing options were available, researchers would find it easier to suggest provisional ideas while being honest about areas of uncertainty.


The current publishing landscape doesn’t offer many choices on either end of this hierarchy. Very few published studies try to formally replicate established results: studies in several fields find that these constitute less than 0.3% of published papers. Similarly, there aren’t many venues to publish nascent theoretical ideas or preliminary findings. In effect, this means that scientists are only incentivised to produce the intermediate-certainty work that tends to be published in journals today. This comes at the expense of many other kinds of valuable science.


Building new institutions


I’ve focused this essay on the publishing model, but as I hope has become obvious from the above sections, we can’t divorce scientific papers from the institutions that produce them. To make different types of publications in the real world, we would need to build new institutions that suit each kind of work.


The organisations that could generate high-certainty publications would probably look very different from traditional academic labs. Systematic replication efforts will need efficient, large-scale work by scientists who are largely disinvested in the relevant hypotheses; it is unlikely that most academic labs would be well equipped to undertake such work.


Instead, replication studies are probably best undertaken by focused organisations that can leverage resources and technical expertise at scale to reproduce the most important findings. It’s good to bear in mind that the converse is also true—such an organisation is probably too inflexible to do the exploratory research done by academic labs. A diverse ecosystem of publications thus requires a diverse ecosystem of research organisations.


Replication work could also be integrated into the academic infrastructure. Part of a scientist’s training, for instance, could include replicating landmark findings from the literature. If these projects were collected in a database, we might eventually accumulate useful information about the robustness of different results. In cases of controversy within a discipline, adversarial collaboration—in which scientists with opposing views co-design and publish experiments to resolve their differences—can be a productive approach. Working together with independent organisations dedicated to replication could make such collaborations much easier.


Theory work would also probably be best undertaken in differently structured environments. Theoretical institutes exist in some fields—these include the Institute for Advanced Study, All Souls College, and the Santa Fe Institute. It would be very useful to find ways to adapt these models for experimental and lab-oriented disciplines. In biomedicine, for instance, theory-driven scientists may work in close collaboration with technically-driven scientists, or otherwise may leverage commissions from Contract Research Organisations.


These new organisations would obviously need sources of funding. One option would be to divert some existing government and philanthropic funding to doing these alternative forms of research—I think there’s enough evidence to suggest that experiments in this space are likely to be in the public interest. Given the huge market for academic publishing, there are also likely to be opportunities for self-sustaining private companies. A mixture of public and private sector organisations would make a new publishing ecosystem more sustainable.


I’d view the need for new organisational structures as an opportunity rather than a drawback. At the moment, 50% of academic scientists quit within five years, and only 3.5% of UK PhD students secure a permanent research position at a university. Different organisational structures will open up new and different kinds of jobs, meaning that more people can find work that suits their skill set.


I’m also hopeful that a greater variety of publishing formats might have positive effects on the incentive landscape. Long-term incentive alignment is a difficult problem for any institution, but different types of publication would at the very least lead to more varied incentives that might help to balance each other out. For example, having results validated externally might come to be a source of prestige for basic researchers, encouraging better scientific practices.


Conclusions


In this article, I’ve argued that moving beyond the traditional journal article can help communicate and incentivise more accurate science. I’ve further contended that seeking to diversify the publishing model by creating different types of publication with different epistemic standards is a better approach than any one-size-fits-all model. Expanding the publishing ecosystem beyond academic journals could help the reproducibility, creativity, and accessibility of scientific discoveries.


There are positive developments in this area—registered reports, pre-print servers, and unconventional science journals could all help to enrich the publishing ecosystem. Making these new formats won’t be easy, but could pay big dividends. Scientific knowledge is critical for many sectors of our economy. Good publishing institutions could give people access to the newest scientific results faster and with greater certainty. This could help to de-risk technological innovation, ultimately leading to new technologies that benefit society as a whole.


143 views0 comments

Comments


bottom of page