The World Health Organization (WHO) has identified over 200 covid 19 vaccines under development as of mid-November 2020. Dozens of them are already being tested in humans in clinical trials. A few pharmaceutical companies have put out press releases announcing that their vaccine is highly "effective" and relatively safe. However, as of mid-November 2020, no detailed results have yet been released that would make it possible to assess the quality and relevance of the results they claim.

When we finally have more than headlines to go on, what questions will need to be asked about these clinical trials? The answers to these questions will help determine the usefulness of these vaccines and define the vaccination strategy, taking account of the interests of vaccinees and society as a whole.

Questions concerning the objectives of the clinical trials of covid-19 vaccines

Vaccination against covid-19 has a variety of objectives. It should reduce the risk of developing the most serious consequences of the disease, such as death or the need for intensive care, reduce long-term sequelae, and reduce the transmission and prevalence of the virus.
A number of questions must therefore be asked about the objectives of a trial to evaluate a vaccine before its results can be properly appraised. Was the objective to detect a reduction in:

• mortality due to covid-19? In the short or medium term?
• the number of patients who required intensive care?
• the number of patients hospitalised for covid-19?
• infections, whether severe or mild?
• the prevalence of the virus, by identifying asymptomatic infections?

The objectives selected determine the numbers and characteristics of the volunteers to be enrolled in the clinical trial, the outcomes to be measured, the data collection methods, the duration of the trial, and so on.

Questions concerning trial design

When evaluating the ability of a covid-19 vaccine to prevent death or reduce the incidence of Sars-CoV-2 infection, the most robust evidence on efficacy will be generated by comparative randomised trials. This means that participants are assigned to the vaccine group or the other group at random, so that the main difference between the groups being compared is whether or not they received the covid-19 vaccine. While the trial is in progress, it is better if participants do not know whether they received the vaccine, in case this knowledge affects their behaviour, such as their adherence to distancing and hygiene measures. It is also better if the investigators do not know either, in case this knowledge affects their evaluation of participants’ symptoms. This is referred to as a "double-blind" trial.

To determine whether a trial is capable of providing answers about the efficacy of a covid-19 vaccine, the questions to ask about its design are:

• was the trial randomised?
• did the participants in the control group (i.e. those who did not receive the covid-19 vaccine) receive a placebo, a different vaccine or no intervention at all?
• was it a double-blind trial?
• which endpoints were chosen (i.e. what outcomes were measured)?
• who were the trial participants? What was their level of exposure to Sars-CoV-2? Did they have close contact with patients infected with the virus, for example healthcare professionals? Was the trial conducted while the prevalence of the virus was high or low? Or were participants deliberately exposed to the virus?
• did it include sufficient numbers of people at increased risk of developing severe forms of covid-19 (due to their advanced age, history of heart or lung disease, diabetes or obesity) to allow interpretation of the data obtained in these groups?
• was the vaccine evaluated in populations that typically have a lower than average response to vaccination, such as very old people or immunosuppressed patients?
• for how long were trial participants monitored? Have plans to release other results, with longer follow-up, been announced?

Questions concerning likely adverse effects

A vaccine must do more than just reduce the incidence and complications of infection: it must also cause less severe health problems than the infection does. Questions must therefore be asked about the vaccine’s adverse effects:

• what type of vaccine was evaluated: attenuated virus, inactivated virus, viral protein, or intracellular insertion of genetic material? Is there a long history of safe use of this type of vaccine? What are the known and foreseeable adverse effects of this type of vaccine?
• does the vaccine contain an adjuvant? Has this adjuvant already been used in other vaccines with known adverse effects? Or has it never been used before, meaning that its adverse effects are highly uncertain?
• do the published results of the clinical trial include detailed data on the vaccine’s adverse effects?
• has longer-term follow-up of trial participants been arranged? In how many participants and for how long? How will serious events be identified?
• were pregnant women exposed to the vaccine? How many? And how did it affect the outcome of their pregnancy?

Questions concerning data transparency

Questions must also be asked about the reliability and transparency of the results presented:

• was the trial recorded in an international registry before enrolment of the first participant?
• was the trial protocol made available before the trial began? Was the trial conducted in accordance with the protocol? Were any changes to the protocol described and were they justified?
• are the published results sufficiently detailed? Were they published in a peer-reviewed scientific journal?
• is the full data set available to researchers who were not involved in the trial, to enable them to verify the quality of the data analysis?
• who funded the trial?

The covid-19 pandemic is an exceptional situation, one of the consequences of which has been the development of vaccines in record time. The urgency of the situation is no reason to hold the evaluation of these vaccines to lower standards. We shall continue to monitor this potential risk.

©Prescrire 20 November 2020

Sources:

• "Évaluation des vaccins covid-19 : des questions à se poser" Application Prescrire 20 November 2020.
• WHO "Draft landscape of COVID-19 candidate vaccines" 12 November 2020. > HERE
• FDA "Development and licensure of vaccines to prevent covid-19. Guidance for industry" June 2020. > HERE
  
• Lurie N et al. "The development of COVID-19 vaccines. Safeguards needed" JAMA 2020; 324 (5): 439-440. > HERE
  
• Doshi P "Will covid-19 vaccines save lives? Current trials aren’t designed to tell us" Br J Med 2020; 21 October 2020: 4 pages. > HERE
  
• Doshi P "Covid-19 vaccine trial protocols released" BMJ 2020; 21 October 2020: 2 pages. > HERE
  

For more information:

• "Covid-19: Follow Prescrire's independent, evidence-based analysis of the pandemic" > HERE