What Makes a Research Study Valid

Darlene Kelly, MD, PhD; Science and Medicine Advisor, The Oley Foundation

The LifelineLetter and other periodicals often report the findings from medical research studies. When deciphering the results, consumers should be attuned to the study design before making any conclusions about whether a therapy is beneficial, better than no treatment at all or better than a previously used therapy. Several factors that go into the design of research can make the results more or less convincing.

Retro or Not?

Let’s say we are looking at a study that tested the effectiveness of drug A versus the standard treatment, drug B, for asthma. Patients that received drug A were considered the "test" group, and those that received drug B, were the "controls." First, there is the issue of whether the study was prospective or retrospective. A prospective study is one that is planned before the data are collected. It offers the advantage of allowing the investigator to organize a study that makes the testing phase and the control phase totally comparable. Using our example, the investigator can be sure that patients who received drug A are comparable to those who received drug B, and that they received the treatment under the same conditions.

By contrast, a retrospective study looks at information that was collected in the past, often from routine clinical laboratory tests. Outcomes based on data collected retrospectively may be inaccurate, because the investigators (and you, the reader) cannot be sure that conditions were equal for the patients being studied. Getting back to our example, if it were a retrospective study, data from the test group who are given drug A today, may have been compared to data collected from the control group who used drug B five years ago. The investigator may have overlooked factors, like air quality, that could have changed significantly during the five year gap in testing. Additionally, patients in the two groups may not have the same type of asthma and the same degree of disease severity.

Are Subjects Randomized?

Secondly, the issue of randomization comes into play. How have subjects for a study been picked for drug A or drug B? Are the participants assigned to a treatment based on a randomization schedule or is the investigator looking at each subject and deciding which treatment they should receive? If the latter plan is used, the opportunity for bias is all too obvious. For example, drug A may be given only to patients with mild asthma, and drug B to those with more severe symptoms; with these unequal conditions, you couldn’t know whether it was the drug or the milder symptoms that produced better outcomes. In a true, randomized study there is typically a scheme for the randomization, and often someone other than the investigator decides who gets which therapy.

Is the Study Double Blind?

A third, related issue is whether the research is what we call "double blind." In a double blind study, neither the investigator nor the subject knows which treatment is being provided. This prevents the investigator (and the subjects themselves) from influencing the subject’s response to the treatment and also prevents the investigator from looking harder for subtle changes based on the therapy. Both of these factors make the data collected from a double-blind research study more convincing.

When reading research that evaluates a treatment’s effectiveness, you’ll also want to know whether the investigator has ruled out the "placebo effect." This is when a subject perceives a benefit from a treatment just because "something" is being done. If the research compares the outcome from patients given treatment X to those given no treatment at all. The individuals who are to receive no treatment at all should actually be given a placebo (such as a sugar pill) that looks and tastes essentially the same as treatment X. This avoids subjective changes that might occur because "something" is being done.

Well-designed research that addresses the issues mentioned above, is often referred to as "Prospective, Randomized, Double Blind" and offers the most convincing results. Sometimes investigators are not able to use these study methods, and various other designs are reported. The results from these studies should be looked at carefully and with a certain amount of skepticism. Some of the questions that you, as a reader, should consider, include: Were all subjects comparable? If there were opportunities for bias on the part of the investigator, was care taken to avoid this? Was the number of subjects large enough to make the conclusions that were made? Was the paper adequately reviewed? (Journals that require "peer review" — where independent colleagues in the same field determine whether the study results are substantiated by the data — offer a more reliable source of research information than journals that don’t.)

Finally, do not hesitate to share research articles with your physician or other health care practitioner. They are valuable resources for helping you evaluate the validity of a study, and equally important, whether the information applies to your personal situation.

Editor’s note: Medical-related information published by the Oley Foundation is reviewed by Oley’s Medical and Research Director and/or other clinicians; however, the Foundation strongly urges members to carefully evaluate any information with their own physician, before making any changes in their care.


updated July 2015