SYSTEMATIC REVIEWS IN ICU: 2003-2018

PLEASE let me know if you see any errors and I will try to fix them ASAP.
HD Fullscreen of the infographs are at the bottom
PDF at bottom of page

I preface this infographic with the caveat that it is an exercise in discussion only, not an exhaustive or conclusive analysis of all systematic reviews within this time period. I have done my best to accurately analyse the trawled data - please do get in touch if you have any suggestions on improvements. I have no conflicts of interest and am not involved in any systematic reviews or trials with any organisation.

£5,774,000 was made as income by the Cochrane Library (in 2016) for publications of Systematic Review. Their total income peaked at £6,805,000 for 2016 as well as them plunging a lot of money (£8.57M) into investing in their 'strategies for 2020'. (See here) This is a big organisation with a lot of clout in the academic world. They had over 9.7 Million downloads of their PDF's (all specialties) in 2016 - I'll be very excited to read this year's report. They have been heralded for years as the barn-door staple of good evidence and systematic review. Hammered into you in medical school, and used by you in the hospital.

My query was; in the past few years, the twittersphere of critical care has been lamenting the era of the 'negative' trial (I hate the term, but we'll leave it at that for now).  I wondered, knowing that so many trials were at best inconclusive or lacking data that could definitively tell us 'if something works' - did systematic reviews, over a period of 15 years in intensive care medicine shed any light as to the quality of evidence of some of these trials, when viewed in the bigger picture of a rigorous systematic review. Moreover, was the data being produced by ICU trials actually allowing us to conclude anything concrete when aggregated.

----------------------------------------------------------------------------------

This is an informal review of 71 Systematic Reviews of topics in ICM that were obtained as follows:

PubMed Search for Cochrane Database Systematic Reviews (SR's)
[ICU or ITU or Critical Care or Intensive Care] in Title or Abstract
Removed all non-related SR's
Removed older versions of SR's that had been re-worked with new articles added.
Only from the COCHRANE database of systematic reviews.

1. Recorded what the abstract detailed as the 'Quality of Evidence'
2. Recorded any 'difference' seen in the study
3. Recorded any clear recommendation given by the Systematic Review (and marked if unclear)
4. Almost all of the studies called for further research
5. Abstracts can be seen within the main document, as well as authors and dates of publication
6. A summary document, for ease, is included
7. You can see all of these trials via PubMed or CochraneLibrary.com

EVIDENCE QUALITY
You can learn about the 'GRADE' framework of scoring evidence quality here.
19/71 = I was unable to assess with GRADE (many hinted at 'LOW' quality evidence and one stated, "adequate") (26%)
31/71 = Very Low to Low (43%)
1/71 = Low to High (1.4%)
4/71 = Very Low to Moderate (5.6%)
5/71 = Low to Moderate (7%)
4/71 = Moderate (5.6%)
2/71 = Moderate to Good (2.8%)
1/71 = Good (1.4%)
1/71 = Moderate to High (1.4%)
3/71 = High (4.2%)

To make it clear, it appears that if we look at the GRADE of evidence as being 'Very Low to Low' the authors had little to limited confidence in the effect estimate reported by the RCT's involved in the trial.

More worryingly - if you look at the abstracts of the papers that I have included in the visual abstracts below, you will see a number have been downgraded in evidence quality via their risk of bias.

Recommendations Given:
4/71 = Yes, No Support
5/71 = Yes, Support
11/71 = Unclear (NO clear recommendation)
51/71 = No recommendation

(We must bear in mind that there is not always an onus on authors to provide recommendations; but when there is clear, strong evidence; it is usually stated within Cochrane Systematic Review.)
​
Discussion:
There is quite clearly, a vast amount of systematic reviews that have been based on trials that the authors have concluded are from very low, low or 'unable to assess' quality of evidence datasets. At best, some of the other trials are from a 'low to moderate' quality background. Authors of many of the SR's declared there were significant biases identified within the component trials (see the abstracts).

We see 'very low or low' quality of evidence side-by-side with lack of differences (due to the quality or bias) being seen and therefore lack of recommendations, which, in fairness, mostly fall outside the scope of an SR -but Cochrane have been 'recommending' or 'not recommending' approaches for years based on SR's.

It is of great importance to view recommendations from systematic reviews alongside their evidence quality category. If 'poor' results go in, 'poor' results are synthesised. One of the more interesting considerations was made by Douglas Altman in the BMJ in 1994; when he says, "much poor research arises because researchers feel compelled for career reasons to carry out research that they are ill equipped to perform, and nobody stops them."

Poor research begets poor research. We can see, with the research in this article, that if you feed very low and low GRADE-marked evidence into a systematic review, if fails to give any useful conclusions (affirmation or negation) and gives rise to requests for 'more evidence' in the forms of a 'high quality RCT'

I think I would feel differently if we have 71 'negative' outcomes (as in - This Does Not Work and there's high quality evidence to show this) vs what we have at the moment, with far too many ICU trials being left out or classified as 'very low' quality due to bias, heterogeneity, poor methodology or other issues. 

Overall, as this is a purely observational exercise, this should make us consider strongly and investigate further:
1. The position of systematic reviews in the evidence pyramid.
2. What ICU trials actually would suit a systematic review by Cochrane?
3. What can trials do to improve?
4. Why have we have not seen a significant trend towards better quality in data/trials over a 15 year period leading to better insights from Systematic Reviews in intensive care medicine.
5. Is it worthwhile to continue performing systematic reviews in this way - should we do something else?
6. Does intensive care medicine inherently suit RCT's and systematic reviews?

Lots of questions - few answers.

FULLSCREEN (HD) VERSIONS:
SUMMARY FULLSCREEN HERE
PARTS 1-2 (Full Abstracts/Authors etc) HERE
PART 3 (Full Abstracts/Authors etc) HERE

PDF Download to come in due course.