Did a change in Nature journals’ editorial policy for life sciences research improve reporting?

Background

Few articles describing in vivo research report taking specific actions designed to reduce the risk that their findings are confounded by bias,¹ and those that do not report such actions give inflated estimates of biological effects.^{2 3} Strategies and guidelines which might improve the quality of reports of in vivo research have been proposed,^{4 5} and while these have been endorsed by a large number of journals there is evidence that this endorsement has not been matched by a substantial increase in the quality of published reports.⁶ Poor replication of in vivo and in vitro research has been reported,^7–9 and this has been attributed in part to poor descriptions of the experimental and analytical details.

In May 2013, Nature journals announced a change in editorial policy which required authors of submissions in the life sciences to complete a checklist, during manuscript revision, indicating whether or not they had taken certain measures which might reduce the risk of bias and to report key experimental and analytical details, and in their submission to detail where in the manuscript these issues were addressed.¹⁰ The development of this checklist was prompted in part by a consensus statement⁵ setting out key aspects of study design and conduct which were necessary to allow the reader to assess the validity of the findings presented; it identified these as randomisation, blinding, sample size estimation and data handling (the ‘Landis 4’). The Nature journals’ checklist also included items relating to figures and statistical representation of data; reagents used; species, strain and sex of experimental animals; reporting of relevant ethical approvals; consent (for research involving human subjects); data deposition; and availability of any bespoke computer code. The full checklist is given in online supplementary appendix 1.

The aim of this study was to determine whether the implementation of this checklist for submissions has been associated with improved reporting of measures that might reduce the risk of bias. Because the Landis consensus statement drew attention to randomisation, blinding, sample size estimation and data handling as being the most important items to report, we chose the reporting of these as our primary measure of outcome. To establish whether any observed change in quality was simply a secular trend occurring across all journals, we matched each included publication with a publication in a similar subject area published at around the same time by a different publisher.

Methods

The methods are described in detail in the published study protocol,¹¹ and the data analysis plan and analysis code were articulated prior to database lock and registered on the Open Science Framework (DOI: 10.17605/OSF.IO/HC7FK). The complete study data set including PubMed IDentifiers (PMIDs) and data descriptors (but not, for copyright reasons, the source pdfs) of the included articles is available on Figshare (10.6084/m9.figshare.6226718).

In this observational cohort study, we aimed to determine whether the implementation of a checklist for submissions has been associated with improved reporting of measures which might reduce the risk of bias. To do this we assessed whether—in the view of trained assessors—manuscripts reported the details required by the checklist. Importantly, we did not have access to the checklists completed by the authors.

The study populations comprised (1) published articles accepted for publication in Nature journals which described research in the life sciences and which were submitted after 1 May 2013 (when the mandatory completion of a checklist at the stage of manuscript revision was introduced) and before 1 November 2014; (2) published articles accepted for publication in Nature journals in the months preceding May 2013 which describe research in the life sciences; and (3) articles from other journals matched for subject area and time of publication. We measured the change in the reporting of items included in the checklist.

Results

Eight hundred and ninety-six articles were identified and uploaded for outcome ascertainment, 448 in each cohort. Two non-NPG articles were excluded because they did not meet the inclusion criteria, and we identified four NPG and nine non-NPG articles which had been included more than once. Four hundred and forty-four NPG articles and 437 non-NPG articles underwent outcome assessment. One NPG publication and one non-NPG publication were adjudged at the time of outcome assessment to report neither in vivo nor in vitro research, and so were excluded. The analysis is therefore based on 443 NPG articles (219 before and 224 after 1 May 2013) and 436 non-NPG articles (194 before and 242 after 1 May 2013) (figure 1). The difference in numbers for NPG and non-NPG before and after 1 May 2013 is because some of the NPG ‘before’ articles matched best with articles in other journals published in the few months following May 2013. Specifically, 26 NPG preintervention articles were matched with other papers published an average of 3.2 months after May 2013 (maximum of 8 months), and 6 NPG postintervention articles were matched with other papers published 1, 2, 9, 11, 12 and 215 months before May 2013. Overall, 43% of matched pairs had dates of publication within 1 month, 54% within 2 months, 64% within 3 months and 81% within 6 months of each other (range −11 to +22 months). Two hundred and thirty-nine articles described only in vivo research, 133 described only in vitro research and 507 described both. Four hundred and ninety-four papers were completely matched for in vivo and in vitro status, 276 were partially matched (one member of matched pair reporting in vivo and in vitro research, the other reporting only in vitro or only in vivo research) and 36 were mismatched (one reporting only in vivo research, the other reporting only in vitro research). The source journals are given in table 1; in total 198 different titles contributed matching articles (median of 1 article per source journal, range 1–47). The PMIDs of included articles are listed in the data supplement.

We had intended to perform subgroup analyses in groups defined by country of origin, categorisation of research, and whether the study was predominantly in silico, in vitro, in vivo or involved human subjects. However, because for some countries the number of included manuscripts was low, the categorisation of research was not available for all manuscripts in the matching non-NPG group (except by inference from the matched NPG papers), the number of predominantly in silico and predominantly human studies was low, and we were not confident that we could operationalise a judgement as to whether a paper was predominantly in vivo or in vitro, we elected not to pursue these analyses. The country of origin for papers in each cohort is shown in table 2.

Two hundred and five individuals registered with the project, of whom 109 started at least one training manuscript, 38 completed their training and 35 assessed at least one manuscript. Of these 35, 12 individuals also reconciled conflicting outcome assessments, and the web interface was programmed to ensure that they were not offered for reconciliation articles that they had previously adjudicated. Including reconciliation, the median number of articles scored was 13 (range 1–441). The agreement between outcome assessors ranged from being no better than chance at 50% (in vivo studies; implementation of statistical methods and measures: ‘Is the variance similar (difference less than two-fold) between the groups that are being statistically compared?’) to 98% (in vivo studies; ‘Does the study report the species?’). The median agreement was 82% (IQR 68%–89%). Two articles were identified during manuscript preparation as having been incorrectly recorded at data lock as reporting both in vivo and in vitro research, where in fact they only reported in vitro research, and one article had been incorrectly recorded as reporting both in vivo and in vitro research, where in fact it only reported in vivo research.

Discussion

The change in editorial policy at NPG was associated with improvements in reporting of randomisation, blinding, exclusions from analysis and sample size calculations. For the highly challenging primary outcome measure, full compliance increased from 0% to 16%. This falls short of the target compliance of 80%, but should be seen in the context, first, that only 1 of 1073 articles from 2009 to 10 from leading UK institutions achieved this standard,¹ and second that overall compliance of 80% would require compliance with individual items of around 95%. Since these results were first communicated at the Peer Review Congress, in bioRxiv and in the peer review process, it has been suggested that the observed change is small. It certainly falls well short of the target of 80% compliance, but this needs to be seen in the context, first, of the generally poor reporting of biomedical research (as evidenced in the non-NPG cohort) and in the challenge involved with achieving compliance for a composite outcome. If the probabilities of compliance with each of the Landis 4 items were independent of each other, then achieving 80% compliance overall would require compliance with individual items of 94%.

Prior to the study we identified achievement of 80% compliance, or an absolute improvement of 15% in the reporting of an item, as being the minimal change which would represent an important effect of an editorial intervention. In the NPG cohort, for 62 items the 95% CIs of the observed change fell below 15% for 11 items, included 15% for 40 items and were above 15% for 5 items. For three items there were insufficient data to calculate 95% CIs, and for three items baseline performance already exceeded 85%.

Power calculations in primary research are often considered unfeasible, on the basis that prior to doing the study the effect size is not known. Our approach here—of identifying a smallest effect size of interest—is increasingly widely used, and has allowed us to demonstrate if any change observed might be as large as the smallest effect size of interest, is definitely that large or is definitely not that large. We hope that those using our findings to guide their own improvements will find this helpful and recommend the approach for use in future studies.

It is notable that even with considerable investment in designing and implementing a checklist, and working with authors to encourage its completion, compliance remains so low. This stands rather in contrast to the belief that ‘all’ that is required to ensure transparency in reporting is that journals ‘insist’ that authors do the right thing. Securing transparency in research reports is a complex challenge, and experience in other fields (MM is also clinical lead for a clinical neurology service) suggests such challenges require a range of complementary approaches with commitment from all stakeholders, might best be achieved through formal improvement activity, and often take multiple attempts to achieve and sustain change.

The checklist relates to transparency in reporting, and articles were judged to be compliant if they either reported measures to address that risk of bias or reported that such measures were not taken. For each of the Landis 4 criteria, compliance was most often achieved by the authors reporting that they had not taken measures to reduce the risk of bias. While this is not ideal, we believe this represents an improvement, in terms of the usefulness of the research to those who wish to use it, from a situation where these issues are not reported at all.

For reports of in vivo research, compliance for randomisation, blinding, reporting of exclusions and sample size calculations in NPG articles reached 68%, 63%, 31% and 64%, respectively. For non-NPG articles the performance was 12%, 5%, 12% and 3%. The figures for NPG articles are similar to those recently reported for in vivo research published in the journal ‘Stroke’,^{13 14} which began requiring reporting of such details following the publication of good practice guidelines in 2009,¹⁵ and where performance was found to be substantially higher than for in vivo research published in other American Heart Association journals.¹⁴

While we saw improvements in the transparency of reporting, the observed improvements in experimental design were much more modest. However, peer review may not ensure the quality of published work,¹⁶ as evidenced for in vivo research by poor reporting of measures to reduce risks of bias.¹ We believe that the ultimate responsibility for assessing research quality (and therefore the validity of the findings presented) rests with the reader, and transparency in reporting is fundamental to this assessment.

For reports of in vitro research, compliance was substantially lower. There have been few systematic attempts to measure the quality of reporting of measures to reduce the risks of bias in vitro research, and our findings suggest that, both in NPG and non-NPG journals, this remains low. There were improvements in reporting randomisation, blinding and sample size calculations in NPG descriptions of in vitro research, but only to 18%, 23% and 34%, respectively. For non-NPG the equivalent figures were 3%, 1% and 1%. There were no significant changes in the reporting of exclusion of in vitro data, with postintervention compliance of 16% in NPG articles and 6% in non-NPG articles.

For other checklist items, changes in performance were less dramatic, but there appeared to be incremental improvements across most of the items measured, although few of these breached our rather parsimonious adjustment for multiple testing. In spite of substantial attention given to the importance of reporting the sex of experimental animals, this was only done in 52% of postintervention NPG studies and in 36% of non-NPG studies.

Our assessment of compliance with the checklist was based on the resulting manuscript, and not on the completed checklists submitted by the study authors, which were not available to us for analysis. Therefore, we do not know whether these submitted checklists were incomplete but the requirement for compliance was not completely enforced, or if the authors and editors considered that manuscripts were compliant but our outcome assessors disagreed with those judgements. Knowing the relative contribution of these two explanations would inform refinements to checklist-based strategies to improve reporting. Of note, for the Consolidated Standards of Reporting Trials checklist, Blanco and colleagues recently showed that the checklist as submitted was concordant with the manuscript as published for only one of six studies.¹

Ours is an observational study, and it is possible that other (related or unrelated) changes were responsible for much if not all the differences seen. These changes were not observed in other journals (at least not when taken in aggregate), and so it is likely that alternative causal factors would relate to NPG editorial policy and practice. While we are not aware of any other relevant changes in editorial policy occurring at a relevant time, it is likely that this change in editorial policy was accompanied by increased attention given to the importance of the quality of reporting by both inhouse editorial staff and external peer reviewers. It is not possible to determine whether these might have caused the changes seen. However, a randomised controlled study of the effect of the Animal Research: Reporting of In Vivo Experiments (ARRIVE) checklist completion on the quality of reporting of in vivo research at PLoS One will report shortly.

While our primary outcome measure was unchanged, when writing our data analysis plan (and prior to any data inspection or analysis), we did change our criterion for measuring success, from ‘whether compliance (with the Landis 4 criteria, for in vivo research) in the postintervention group of articles reached 80%’ to ‘the change in proportion of articles describing in vivo research meeting the 4 Landis criteria’. This was because our primary intention had been to observe any effect of a change in publication policy, and with the benefit of hindsight this was not captured in our original primary outcome, but we recognise this as a limitation in our findings. We note, however, that the primary outcome used reflects better the title of the study protocol than does the primary outcome measure proposed in that protocol.

For our comparator group we chose similar articles with a similar date of publication identified using the PubMed ‘related citations’ tool. The journals in which these works were published will vary in the attention which they have given to transparency in reporting, and it may be that for some journals there have been changes similar to those observed in the NPG articles. While we might have restricted our comparator group to journals more similar to NPG articles (for instance by impact factor or extent of editorial intervention), this would have meant lower fidelity of matching by subject area or date of publication or both, and we considered these factors to be more important. For this reason, our findings for NPG articles cannot be interpreted as showing improved reporting compared with similar articles in similar journals. The representation of such ‘similar’ journals in the comparator group is too small to allow meaningful conclusions to be drawn.

During the study we encountered some difficulties that we had not expected. We had thought that it would be straightforward to distinguish between an in vivo experiment and an in vitro experiment, but we had to develop an operational approach which defined that experiment on the basis of the subject at the time that the experimental intervention occurred; so a tissue slice experiment involving tissues from animals exposed to treatment or control we considered in vivo, while a similar experiment applying drugs directly to the slice we considered to be an in vitro experiment.

Our matching on whether studies reported in vitro or in vivo research or both was also reasonable in most cases. Differences will have emerged where, as described above, articles were initially categorised with one set of characteristics (in vitro, in vivo or both) and matched accordingly, but later judged to have different characteristics. Our matching for date of publication worked reasonably well, apart from the inclusion of one comparator article published in 1995, 215 months before its ‘matching’ NPG article. We had not anticipated that matching articles would be so difficult to identify, so our matching rules did not have an upper limit of difference in the date of publication. An alternative approach would have been to prioritise matching on data of publication rather than manuscript content, but each approach has its weaknesses. Because one group (non-NPG studies published before May 2013) is substantially smaller, this will have limited, to an extent, the statistical power of these contrasts; however, since power changes with the square root of the number of studies, we estimate this loss of power only to be around 10%. Since the comparator (non-NPG) group does not contribute to our primary outcome, and the matching is generally good, we do not think that these mismatches devalue our findings to any appreciable extent.

Our matching by country of origin in the NPG cohort of publications may have introduced a bias in that manuscripts from countries with fewer publications may have been excluded because of the lack of an appropriate match. However, the included manuscripts had a country of origin matching more than 85% of Nature papers published between 2010 and 2016, and so it is unlikely that this has introduced major bias.

Further, there were some checklist items where agreement between outcome assessors was very low—for instance, for the question of whether for in vivo research the difference in variance between groups being compared was less than twofold, the agreement was no better than would be expected by chance alone. We recommend that the future development of publication checklists should include an assessment of interobserver variation by potential users of the checklist for each checklist item; low agreement might indicate that the item should be rephrased or reframed, or that more explanatory text is required.

We encountered a further unexpected problem when assessing compliance with reporting of blinding, randomisation and with sample size calculations. These were assessed with pairs of questions: first did the study report doing it (yes/no/not relevant); and second did they at least mention it (yes/no/not relevant). If a study was ‘yes’ for the first question, assessors were instructed to score the second as not relevant. Therefore, the number scored as ‘not relevant’ for the second question should represent the sum of those scored as ‘yes’ and as ‘not relevant’ for the first. This was not always the case (for in vivo research occurring in 0.1%, 0.8% and 6% of assessments for sample size calculation, randomisation and blinding, respectively), but we did not become aware of this problem until after database lock. Any impact of this shortcoming is likely to be small.

Finally, our work shows the challenge of assessing even a relatively limited number of articles against a relatively straightforward checklist. We are delighted that so many collaborators (from six continents) agreed to participate and are very grateful to them. However, even with their help the outcome assessment and reconciliation took 17 months. This is too slow to be useful, for instance, for quality improvement activity, where more rapid feedback would allow more rapid adjustments in response to performance. We have tested the use of text analytics using regular expressions to automatically ascertain reporting of measures to reduce the risk of bias, and for some such risks of bias the approach achieves sensitivities and specificities above 80%.¹⁷ For more complex items it is likely that machine learning approaches using, for instance, convoluted neural networks may be more successful, and this is a current focus of our research. We hope that, by making the data set for this study available, this might be used, for instance, for distant supervised learning in such systems. However, the extent of disagreement between our trained assessors suggests that the language used to describe experiments in biomedicine is not altogether clear, and both machines and human may require greater clarity in reporting to fully understand published research.

Conclusions

Introduction of a checklist leads to substantial improvements in the quality of reporting in NPG articles that were not seen in matched articles from other publishers, and these improvements appear to be ongoing. However, there is still substantial room for improvement, which suggests that measures such as mandatory author checklists need to be supplemented by other approaches.