Erik Goldman03.01.09
Don’t Be So Negative!
Learning to view negative studies in the context of the greater body of knowledge on a particular substance is just what the doctor ordered.
ByErik Goldman
The pattern is almost laughably predictable: epidemiological and observational studies suggest that a particular nutrient or botanical may prevent or ameliorate a common chronic disorder; animal and cell culture studies suggest a plausible physiological mechanism; small clinical studies give encouraging findings, and everyone gets excited that we’ve got a non-drug alternative for a difficult disease.
Then, somebody lands a huge grant from National Institutes of Health (NIH) or some other “Big Science” establishment, and does a full-on, large-scale, randomized, controlled clinical trial. The data come up equivocal at best, negative at worst, and before you know it, the headlines are screaming that this or that simply “doesn’t work.”
In the last few months, the Journal of the American Medical Association (JAMA) has been chock full of bummer trial reports: the combination of vitamins C and E did not reduce cardiovascular disease risk compared with placebo in the Physicians’ Health Study (Sesso HD et al. JAMA. 2008); in the SELECT trial, selenium and vitamin E did not lower prostate cancer risk (Lippman SM, et al. JAMA. 2009); in the GEM trial, Ginkgo biloba failed to prevent Alzheimer’s associated memory loss in the elderly (DeKoskey ST, et al. JAMA. 2008).
What Gives?
Why do the big trials often come up negative when preliminary research looks so positive? Is the epidemiology wrong to begin with? Were the trials improperly conducted? Are the researchers and trial designs biased against natural products? Is it true, as some industry advocates claim, that natural products just cannot be studied in conventional randomized trials? Are supplement critics right in asserting that nutrition-based medicine is little more than pipedreams and marketing hype?
Paul Coates, PhD, director of the Office of Dietary Supplements at NIH, is uniquely qualified to weigh in on these questions. His job is to set the agenda for NIH-funded nutraceuticals research.
“I know some people in the supplement world take umbrage at randomized controlled trials (RCTs). But it is not impossible to do good RCTs with nutrients, and it doesn’t mean that negative results are wrong,” said Dr. Coates in an interview. Indeed, the RCT worked pretty well to document the impact of folic acid in preventing neural tube defects. No one seems to question the study design when the data are positive.
Still, the recent wave of negative studies does raise the suspicion that perhaps we’re not asking the right questions, or that we have unrealistic expectations.
“Epidemiological and observational studies cannot give cause-and-effect proof. They do provide clues about where to look. If the signals are strong enough, those clues should be followed and tested. In these recent trials, it may simply be that the clues we got from the epidemiology and observational trials were not indicative of a strong effect.”
In the case of vitamin E and heart disease, the epidemiological signals came from studies looking at markers of vitamin E in the blood, and comparing the heart disease risk in people with high versus low levels of these markers. This led to trial design around one or two of the possibly relevant nutrients. “We have to recognize that once we move to an intervention design, we cannot include everything that might be relevant.”
In the case of the SELECT trial, the initial impetus for studying selenium in prostate cancer came from an earlier selenium study that did not have prostate effects as a primary outcome; the observation that the mineral might reduce risk was a sort of “by the way” finding, and one that might have been misleading.
Part of the problem in designing nutraceuticals is that researchers, marketers, and the public often expect nutrients or botanicals to behave like drugs, giving big, discrete and easily measured benefits in a broad range of people. But nutrients and botanicals are not pharmaceuticals. Consequently, Dr. Coates thinks we need to adjust our expectations.
Generally speaking, few people in the U.S. have frank nutrient deficiencies (i.e., scurvy, rickets, beri beri, etc.), so supplementation seldom has dramatic “big bang” effects. Using vitamin C as an example, he noted that while many people fail to get optimal amounts, few have scurvy. “If you give a lot of vitamin C to people who are more or less replete, you may not see much effect. The net effect was basically zero in the Physicians Health Study. It’s going to be hard to see a strong signal because the effect size is small to begin with, and the level of noise is high,” said Dr. Coates.
Rolling with the Punches
Jeffrey Bland, PhD, president of Metaproteomics, and founder of the Institute for Functional Medicine, views nutrients as physiological “shape-shifters,” exerting subtle and non-specific effects on multiple physiologic pathways, rather than strong effects on a relatively small number of pathways, which is how pharmaceuticals work. But many of the large-scale NIH-funded trials are premised on simplistic, drug-oriented, single-pathway thinking.
He, and others concerned with nutrition-based preventive medicine, would like to see future NIH trials do a better job of controlling for variables like subjects’ diets, oxidative stress status, and genetic predispositions for various metabolic states. The goal should be to identify which people are most and least likely to respond to specific nutrients. Nutrition and supplementation is not a “one-size-fits-all” proposition.
High-profile government-funded studies carry a lot of weight with physicians. But all too often, busy doctors run with the top-line findings, and miss secondary but important signals. Mary Hardy, MD, medical director of the UCLA Center for Integrative Oncology, notes that while the SELECT study did not bear out the hoped-for prostate protective benefit, it did show that there was no increase in selenium-associated adverse effects after six years of continuous use. That’s an important bit of safety information for anyone taking this mineral for other purposes.
The same holds true for ginkgo. Although DeKoskey’s GEM study showed no obvious benefit in preventing memory loss, it also showed no evidence of spontaneous bleeding, which all but dispels the previously held notion that ginkgo can cause dangerous vascular side-effects.
“We might not have seen an Alzheimer’s preventive effect, but we gained a lot of important safety data for ginkgo. Always read the safety results of these trials, even if the primary results are negative,” Dr. Hardy advised.
Mark Blumenthal, director of the American Botanical Council, said that a big negative study often overshadows a whole body of evidence favoring a particular herb, leading to a warped view of its clinical utility.
“People latch on to the latest study and ignore the totality of evidence. It really irks me to see high-profile people writing editorial pieces maligning a particular herb in major newspapers based on one negative study.” For example, a recent negative study of saw palmetto for prostate problems was the first negative trial out of more than 30 prior studies, all of which were more or less positive. “They took that one trial as sufficient evidence to dismiss a useful herb,” he said.
Back to ginkgo, the GEM findings have all but drowned out an earlier study in which families of patients with Alzheimer’s disease reported consistently better quality of life, using the Clinical Global Impressions scale, in those taking the herb.
“Managing something like Alzheimer’s is not just about memory function. It’s about the ability to get on in daily life, to get along with other people,” said Mr. Blumenthal. “Ginkgo has proven itself pretty well in reducing symptoms of early stage Alzheimer’s, even if it doesn’t prevent the disease. But the public seldom hears about that. They’ll just get a simplistic message that “ginkgo doesn’t work.” There are some very positive conclusions about herbs that often get drowned out by high profile negative studies.”
Dr. Coates said negative studies are a fact of life in science. While it may seem like supplements are being unduly bashed these days, the reality is that pharmaceutical companies often come up empty-handed when they put their products to the test.
“Most things don’t work all the time. In the case of St. John’s Wort, there was a study a few years ago showing that it doesn’t work well in mild to moderate depression. That simply puts it in the company of most antidepressant drugs, which don’t work 50% of the time.”
While Dr. Coates acknowledged the need to move away from simplistic drug-like thinking when studying nutraceuticals, he also stressed that the ultimate goal of the scientific endeavor is to expand our understanding of biology and the ways various substances interface with our physiology, not necessarily to tell us what we want to hear.
“My job is to identify gaps in knowledge and fill those gaps with research. We don’t do studies when we absolutely know what the answer is going to be. That’s not good use of public research funds,” he commented.
Negative studies are part of the nature of the scientific process. Though they may not be helpful in selling products, they can be just as important as positive ones in the evolution of our knowledge.