Survey-based research is sometimes derided as lacking rigor and being too easily steered toward a desired outcome.  But nutrition experts say that when done well it fills a vital role and can help answer some questions that can’t be tackled in other ways.

Some large-scale studies, like the ongoing Nurse’s Health Study, or NHANES study (National Health and Nutrition Examination Survey), are based in part or wholly on the results of surveys. NHANES in particular has provided valuable data sets that can then be analyzed further to provide key insights into various questions of the interplay of diet, nutrition and the long term risk of some diseases.

For these huge, long term studies, the inevitable errors and distortions in the filling out of the surveys are ameliorated by the huge number of replies.

Approach can yield significant contributions

But how about for smaller studies?  How are these viewed within the scientific community in terms of their utility and the validity of the results? Are they just a way to get a quick and dirty result that a brand can use to hawk the purported benefits of an ingredient or finished product?

Experts contacted by Natural Products Insider by email said when done well, these studies are a vital part of the research picture for natural products.

Related:NIH: Omega-3 Supplements Largely Meet Label Claims

Susan Hewlings, Ph.D., vice president of research affairs of Del Mar, California-based Radicle Sciences, said the same strictures apply to these kinds of studies as to others. Chief among those considerations is a double-blind placebo-controlled design.

Special considerations for surveys

In addition, she said there are some special considerations, such as the type of survey questionnaires used. These should be “validity- and reliability-tested,” Hewlings said. Researchers who make up their own questionnaires ad-hoc run the risk of having their results called into question.

(Full disclosure: Radicle Sciences is a contract research organization focused on survey-based studies.)

“There is an overall misconception that surveys are ‘soft science’ or not as rigorous. While, of course, any scientific methodology can be utilized in a non-rigorous fashion, when proper rigor is applied surveys are a valid way to conduct research,” Hewlings said.

“Then you need a proper power analysis with an appropriate effect size, and you must consider your potential attrition rate to make sure that you recruit an adequate number of subjects to maintain the power of the study. This is part of a well-designed statistical plan. The plan is established as part of the study design, not after the fact. The plan should state intentions for outliers, how to handle those that do not complete the study, the statistical analyses and all the details before the study begins. After the fact and post-hoc analyses considered after the results are complete can decrease the rigor of the study,” Hewlings said.

Finding out how subjects feel

One big benefit of survey-based research is it can yield a key result for brands doing research on their products, namely whether the study subjects felt a difference. Research based on blood biomarkers or other similar measures might help sell the first bottle, but if end users don’t feel a difference, it won’t help sell a second.

“Heuristic data is important along with clinical data to tell a story. So just because an ingredient may be metabolically efficacious, if the participant doesn’t recognize a benefit, you might not have a good enough story to tell to sell it. How people feel matters, too,” said Susan Kleiner, Ph.D., a clinical nutritionist who is a founding member of the International Society of Sports Nutrition.

Hard data points can be designed in

Stefan Gafner, Ph.D., chief science officer of the American Botanical Council, said surveys can be designed to get hard data points, too, not just how subjects "feel” about an intervention.

“I looked at the Rivermead mobility index survey. Questions include, ‘If you drop something on the floor, do you manage to walk 5 meters, pick it up and then walk back?,’ or ‘Do you manage a flight of stairs without help?’ So, these ‘yes’ or ‘no’ questions look at measurable endpoints, even if they don’t include physiological parameters such as blood levels of disease markers, or other measurements, such as waist circumference, blood pressure, or pulse,” Gafner said.

Surveys can be a cost-effective way to determine if a study program is going down the right path, Kleiner said. But even then, in her view, it has to be paired with hard data derived from blood assays, biometric measurements and the like.

“I see survey data as enhancing clinical data, not really the other way around. Except when funding is not enough for clinical research and you start with survey data,” Kleiner said.

“If a brand believes that their product improves gut health and they want to say it does so by changing the diversity of the microbiome, then a biomarker will be needed. Of course, there are also circumstances where a biomarker alone can carry the study or the claim. An extreme example would be a claim about rapid absorption—a pk/pd study is needed and a survey won’t add to support for the endpoint,” Hewlings said.

Some questions best suited to surveys

All three experts agreed that there are certain endpoints, such as mood states, libido and others that are only amenable to a survey research approach.

“Pain research is one area where patient surveys are the most useful approach to evaluate the treatment outcome. Anxiety and depression is another set of conditions where patient-based or provider-based surveys are the main way to assess treatment success in clinical studies,” Gafner said.

“Taste is an obvious one. Experiential questions are best for surveys,” Kleiner said.