"Significant"-a gateway to confusion

October 4, 2011

Significance

If there is one word that distorts the meaning of scientific research to the public it is the title of this essay, significant. The difficulty lies in it having two meanings, somewhat similar but quite different. The first, which is the most common usage means "important, having or likely to have influence or effect." The second meaning can actually denote something much less, only that if all of the unknown inputs were controlled that the research shows some difference, even if trivial, caused by what the research is exploring.

The first definition, which all but those who have mastered advanced statistics take the word to mean, can never be derived from any given research, as it relates to values; since saying that something is important is a personal conclusion that is made in the context of the larger issue being explored.

It is in the personal interest of every researcher to depict his or her work as being socially important, as meaningful, to be seen as significant in the first usage of the term. Therefore it is understandable that they will describe the results of their research as showing "significant differences", notwithstanding the knowledge that when communicated to the public it will be viewed as meaningful, even though it may not be.

Usually, if one reads the full press release there will be words of caution such as, "these are only preliminary studies and additional work should be done." Yet, the headlines and the reports that go out to the news services do not include these warnings, and use of the word significant by those from prestigious institutions are taken by readers as much more than the technical findings that allowed the work to be published in a peer reviewed journal. This results in the public facing a barrage of news about their most pressing concern, preserving physical and mental health, that seems to be contradictory and inconsistent.

This recent sentence in an article in the N.Y.Times illustrates this problem, "The study showed that women with higher levels of long-term exposure to air pollution had 'significantly' faster declines in cognition than those with less exposure to pollutants." There is no way to know what the quotations around the word significantly meant. Was it an acknowledgement by the editor that this study may not be meaningful, or is it to accentuate the importance of the finding, that we should all be aware of this unreported danger of pollution.

There is a solution to this problem of ambiguity that underlies much of the public's confusion. The word significant must be removed from the lexicon of science reporting at all levels. Any given study that reports any difference in experimental outcome implies that it is statistically significant, and it's very reporting means that it is not random artifact. The degree of certainty of this is expressed in a P,or probability percentage, which could simply be included as (P=.05). This could be left out of mass media articles, but the word significant would no longer be there to confuse the general reader about the particular finding's importance. In journal synopses and discussions of the research, commonly understood words such as "suggestive," "moderate," or "sizable" can allow the researcher to describe the effect to other researchers and to the public.

Elimination of this inherently ambiguous term would not diminish communication in any way. Actually this would be one of the few examples of an effect opposite to what is called "political correctness" which in the name of social benefit usually sacrifices precision of communication. The enterprise of science, which is based on objectivity of research and of accuracy of communication, must enforce a lexicon that advances these goals. Continuing use of a word that only leads to confusion must no longer be accepted by scientists who generate research, or by journalists who translate their efforts to the public.

--------------

Addenda:

This article expresses the issue I cover, where "Clinical Significance" is the first definition, "important" contrasted with "Statistical Significance" which represents the later, or trivial difference in experimental results.  .

Statistical Significance versus Clinical Significance
 http://www.medscape.com/viewarticle/703355_5

A general issue that has arisen in many of the meta-analyses reviewed is the importance of clearly understanding the differences between statistical significance and clinical significance. The ChEIs and memantine have demonstrated statistically significant improvements in outcomes including cognitive and behaviour when compared with placebo, but how do these improvements translate in a clinical setting? A recent review by Hogan [2007] cautioned that many of the reported results from RCTs may be statistically significant but researchers must go a step further to prove that these benefits are clinically relevant. In order to avoid this exaggeration of results, methods including the reporting of effect sizes, NNT, or setting a minimum change in score in order to be labelled a ‘responder' should be considered [Hogan, 2007].

Articles referenced in OpEd Submission:

Air Pollution Linked to Heart and Brain Risks The study showed that women with higher levels of long-term exposure to air pollution had “significantly” faster declines in cognition than those with less exposure to pollutants.

----  How 1-Minute Intervals Can Improve Your Health Despite the small time commitment of this modified HIIT program, after several weeks of practicing it, both the unfit volunteers and the cardiac patients showed significant improvements in their health and fitness. The results, published in a recent review of HIIT-related research, were especially remarkable in the cardiac patients. They showed “significant improvements” in the functioning of their blood vessels and heart, said Maureen MacDonald, an associate professor of kinesiology at McMaster who is leading the ongoing experiment.

Example of Public Relations announcement that benefits from ambiguity of the word.

-------- EnVivo Pharmaceuticals Announces Significant Results from GSM Transgenic Mouse Model for Alzheimer's Disease In its study, EnVivo was able to show that EVP-0015962 significantly reduced the production of total amyloid load as measured by plaque count, surface area and aggregated AB in the brains of transgenic mice after 12 months of daily oral administration.

A Bill now in the house committee on government affairs, The Research Works Act H.R.3699, would further limit public dissemination of research that is funded by public funds.  A separate essay on this is being written.  This is a summary

http://openaccess.eprints.org/index.php?/archives/867-guid.html
---------------------------
From Journal American Scientist

Do We Really Need the S-word?

The use of “significance” in reporting statistical results is fraught with problems—but they could be solved with a simple change in practice

Megan D. Higgs