The Decline Effect
While there is a complex spectrum of attitudes toward science, there are three clusters worth pointing out, specifically in reference to the provocative New Yorker article by Jonah Lehrer called The Truth Wears Off. The first group are those with an overly simplistic or naive sense of how science functions. This is a view of science similar to those films created in the 1950s and meant to be watched by students, with the jaunty music playing in the background. This view generally respects science, but has a significant underappreciation for the flaws and complexity of science as a human endeavor. Those with this view are easily scandalized by revelations of the messiness of science.
The second cluster is what I would call scientific skepticism – which combines a respect for science and empiricism as a method (really “the” method) for understanding the natural world, with a deep appreciation for all the myriad ways in which the endeavor of science can go wrong. Scientific skeptics, in fact, seek to formally understand the process of science as a human endeavor with all its flaws. It is therefore often skeptics pointing out phenomena such as publication bias, the placebo effect, the need for rigorous controls and blinding, and the many vagaries of statistical analysis. But at the end of the day, as complex and messy the process of science is, a reliable picture of reality is slowly ground out.
The third group, often frustrating to scientific skeptics, are the science-deniers (for lack of a better term). They may take a postmodernist approach to science – science is just one narrative with no special relationship to the truth. Whatever you call it, what the science-deniers in essence do is describe all of the features of science that the skeptics do (sometimes annoyingly pretending that they are pointing these features out to skeptics) but then come to a different conclusion at the end – that science (essentially) does not work.
I often feel that those in this third group – the science deniers – started out in the naive group, and then were so scandalized by the realization that science is a messy human endeavor that the leap right to the nihilistic conclusion that science must therefore be bunk.
But scientists themselves are generally in the middle group (at least they should be), and so news about the various foibles of science are no surprise.
The New Yorker Article
The article by Lehrer falls generally into this third category. He is discussing what has been called “the decline effect” – the fact that effect sizes in scientific studies tend to decrease over time, sometime to nothing. This term was first applied to the parapsychological literature, and was in fact proposed as a real phenomena of ESP – that ESP effects literally decline over time. Skeptics have criticized this view as magical thinking and hopelessly naive – Occam's razor favors the conclusion that it is the flawed measurement of ESP, not ESP itself, that is declining over time. Lehrer, however, applies this idea to all of science, not just parapsychology. He writes:
And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.
This paragraph sums up what I was describing above – Lehrer is ultimately referring to aspects of science that skeptics have been pointing out for years (as a way of discerning science from pseudoscience), but Lehrer takes it to the nihilistic conclusion that it is difficult to prove anything, and that ultimately “we still have to choose what to believe.” Bollocks!
Lehrer is describing the cutting edge or the fringe of science, and then acting as if it applies all the way down to the core. I think the problem is that there is so much scientific knowledge that we take for granted – so much so that we forget it is knowledge that derived from the scientific method, and at one point was not known. Many conclusions in science do not decline over time – they strengthen, and they become so overwhelmingly confirmed that they can be treated as established facts (although always open to revision). The theory of gravity, relativity, evolution, the DNA basis of genetics, the germ theory of infectious disease – these are not illusions that evaporate under closer scrutiny, but fundamental aspects of reality that science has elucidated.
It is telling that Lehrer uses as his primary examples of the decline effect studies from medicine, psychology, and ecology – areas where the signal to noise ratio is lowest in the sciences, because of the highly variable and complex human element. We don't see as much of a decline effect in physics, for example, where phenomena are more objective and concrete.
The Decline Effect
If the truth itself does not “wear off”, as the headline of Lehrer's article provocatively states, then what is responsible for this decline effect? The answer can be found in the previous articles of this blog and many others, like science-based medicine. But to quickly review – it is no surprise that effect science in preliminary studies tend to be positive. This can be explained on the basis of experimenter bias – scientists want to find positive results, and initial experiments are often flawed or less than rigorous. It takes time to figure out how to rigorously study a question, and so early studies will tend not to control for all the necessary variables. There is further publication bias in which positive studies tend to be published more than negative studies.
Further, some preliminary research may be based upon chance observations – a false pattern based upon a quirky cluster of events. If these initial observations are used in the preliminary studies, then the statistical fluke will be carried forward. Later studies are then likely to exhibit a regression to the mean, or a return to more statistically likely results (which is exactly why you shouldn't use initial data when replicating a result, but should use entirely fresh data – a mistake for which astrologers are infamous).
All of these effects, and more, are why skeptics are frequently cautioning against new or preliminary scientific research. Don't get excited by every new study touted in the lay press, or even by a university's press release. Most new findings turn out to be wrong. In science, replication is king. Consensus and reliable conclusions are built upon multiple independent lines of evidence, replicated over time, all converging on one conclusion. But such conclusions are possible, and occur so often we take them for granted. While initial conclusions that turn out to be wrong are sensational – as is turning such events into an indictment of science itself.
Conclusion
Lehrer does make some good points in his article, but they are points that skeptics are fond of making. In order to have a mature and functional appreciation for the process and findings of science, it is necessary to understand how science works in the real world, as practiced by flawed scientists and scientific institutions. This is the skeptical message.
But at the same time reliable findings in science are possible, and happen frequently – when results can be replicated and when they fit into the expanding intricate weave of the picture of the natural world being generated by scientific investigation.
But apparently it is more provocative to focus on the edges of science where results are preliminary and replication calls them into question. It is true that often preliminary results are seized upon, and in medicine even acted upon, before they are adequately confirmed. That is an important lesson. But it does not mean that science doesn't work. It just means that science grinds more slowly than perhaps we would like. Science is self-corrective, and perhaps there is more to correct than we would like. But in the end the process of science works itself out. It is not just another narrative, and we don't have to arbitrarily choose what to believe.
Science nihilists tend to use the “crack-no-wall” argument. That is as long as any flaw exists in an argument it affects the validity of the whole argument. The problem is that most science has problems. Scientists put it into context while nihilists just point out the problems.
This is specially unnerving when applied to important issues that interface science, politics and economics. Scientists become just one side of the issue with equal validity to opinion and ideology. My believe is that an important, if not the role of the modern skeptic, is to educate society on the complexity of science and why if you accept the scientific consensus you are more likely to be right at present and definitely in the long term.
It seems to me that, depending upon the subject of scientific inquiry, one could be a healthy skeptic at one time, naive about another subject, and nihilistic about yet another. I have friends who have graduate degrees, and are very skeptical about research in their given profession, and even with regard to other areas within an academic environment, but toss in a cherished necked, and they become naive, and usually nihilists, when the evidence doesn’t jive with their pet belief.
It seems like these aren’t distinct states, but rather points on a continuum… Credulity/Bias Spectrum Orientation Disorder.
I meant “cherished belief” spell chick will be the death of me… especially if I keep texting while driving!
“The third group, often frustrating to scientific skeptics, are the science-deniers (for lack of a better term). They may take a postmodernist approach to science – science is just one narrative with no special relationship to the truth. Whatever you call it, what the science-deniers in essence do is describe all of the features of science that the skeptics do (sometimes annoyingly pretending that they are pointing these features out to skeptics) but then come to a different conclusion at the end – that science (essentially) does not work.”
You mean like saying DDT is safe and should be used more, which opposes all of the actual scientific research that has been done?
Hey, none of that snark, or I will spray you with DDT to determine if you develop resistance as fast as a mosquito does!
Speaking of …. DDT remains in those farmers’ soils for some time. http://www.hcn.org/blogs/goat/would-you-like-some-ddt-with-that-organic-cucumber
And, here’s a full story on DDT as long-term farmland residue:
http://www.hcn.org/issues/42.21/farmings-toxic-legacy
Forgot that this story is subscription-walled, so I’ll give a brief summary and quotes:
The U.S. EPA does not require sampling of former orchards or other farmland when converted to schools, residential neighborhoods, etc. A couple of Western U.S. states do state level testing, including Washington and California. Findings for one Yakima, Washington family?
I read the article myself a few days ago, and while I also cringed at some of the phrasings (like “we still have to choose what to believe”), I didn’t get the overall impressions of his conclusions being nihilistic or anti-scientific at all. I thought he did a pretty good job in pointing out some problems with the practice of science (maybe well-known to scientific skeptics, but certainly not to many readers of The New Yorker). I don’t think there is a reason to get too defensive about this, even though the usual suspects will certainly spin it into some postmodern anti-scientific nonsense. And by the way, the author clarified some of his statements over at Wired Science: http://www.wired.com/wiredscience/2010/12/the-mysterious-decline-effect/
Here’s a PDF of the article under discussion.
Personally, while I thought there were some flaws in Lehrer’s article, which Steve explains brillitantly in his post, I have to wonder if he read the same article I did. I just don’t see how Lehrer’s New Yorker piece either supports a post-modernist interpretation of science or engages in denialism. In fact, lines like, “This suggests that the decline effect is actually a decline of illusion.”, would seem to suggest otherwise since Lehrer comes to the same conclusion as Steve.
Overall, I think though flawed it was a really good article. We need more of this kind of thing in the popular media precisely because so many people, including skeptics, do have a naive understanding of science.
I agree, I thought it was a pretty good article. After all, it was the New Yorker and it had to be readable from the editor’s point of view…so you weren’t going to see 50 pages of nuanced discussion of the differences between human factors and pure physics.
I think one of his main points, that wasn’t touched on by Novella, was that sometimes the use of statistics (confidence intervals) in early studies can improperly give an impression of an issue being settled. The author implies that this might be gamed a bit by scientists trying to get published. The popular press can then run with the story and suddenly everyone thinks that drinking 13 gallons of guava juice per day will extend his lifespan.
I don’t think that should be a shock to anyone, nor is it an indictment of all science. A recent real-world example of this might be the fanfare surrounding NASA’s announcement about the arsenic replacement in DNA. It seems to me that Novella’s second group of scientific skeptics were the ones most troubled by that performance.
I had read just the precis of the story, via Skeptic’s Dictionary, last week.
That said, via a friend, here’s a Google Docs version of the full deal: https://docs.google.com/leaf?id=1WiT08jJwfMnTyev3ueXd5AtVOOO85iHxtjxfcbkVfHdho2lVi7wOKRSWCKXC
That said, I think this is more than “at the fringes of science.” The decline effect is well-known in psychotropic medications, namely antidepressants, and has been written up numerous times. To quote Lehrer, it’s “occurring across a wide range of fields, from psychology to ecology.” And, in conjunction with many of these fields, he notes publication bias is a partial but not total explanation.
One thing that Lehrer doesn’t discuss in depth is medicine and social sciences’ low P value of 5 percent. Now, in medicine, that’s understandable; one doesn’t want to screen out potentially lifesaving measures. Even there, I think it could be tightened, though, to say 3 percent. It could be tightened even more elsewhere.
We don’t see the same degree, at least, of “decline” issues in physics, certainly.
That said, even in physics, Lehrer does note the element of “chance.” This is the universe, and we’re not its master, not even the small corner called Earth.
However, what does this MEAN? As Lehrer notes, even with variations of more than 2 percent in tests of gravitational strength, we haven’t altered any gravitational theorizing.
What SHOULD it mean? There’s where the rubber really hits the road, and I’d love to see a Part Two to this story from Lehrer.
So, I don’t see “fringe science” here, just because he used Rhine as one of many examples.
The only real criticism I’ve had of Lehrer in the past, speaking of depression, is his skirting with the waters of Pop Ev Psych in claiming depression is likely adaptationist.
And, that said, riffing off Beelzebud’s snark, Pop Ev Psych probably fits right in with libertarianism and hereditarianism/racialism as more serious problems within **alleged** science for Novella to concern him about than Lehrer.
I think what Lehrer’s observation on the element of chance also shows, indirectly, is what David Buller says in his takedown of Pop Ev Psych, and that’s that good scientists still need good philosophers of science working behind the scenes.
I agree that Lehrer was softer in parts of his article, and also in his blog (which I did not discuss) – but I think the article gave an overall nihilistic impression that is not warranted. Spin matters, because that’s what the public mostly comes away with anyway.
You could have discussed the exact same factual material with a completely different emphasis, and there would have been no problem.
“Spin matters”? See “SkepticBlog, post immediately before this one, concerns about possible libertarian ‘spin.'” Wash, rinse repeat.
In other words – there are the facts, then there is the narrative into which you weave the facts. That narrative matters when you are a journalist summarizing a broad range of research for the public. The whole point of articles like that is to construct a narrative (spin) for commentary. What narrative the journalist chooses to tell, matters.
But isn’t that the problem? That journalists, including those covering science issues construct a story device to appeal to the most people possible instead of explaining the facts and their relevance.
The problem is constructing the story device first then backfilling the facts and data. The goal should be to find out what the real story is, based upon the research – and tell that story. Telling science as a story is not a problem in itself – I think that works well.
That’s what happens when English majors do science reporting :-)
Somite, and even more, Steve, that’s also what happens when libertarian bloggers “construct the story device first then backfill,” isn’t it?
Doorknob, I love seeing petard self-hoisting in action.
Well, I was referring to Dunning’s spin, Steve, and NOT the alleged spin of the alleged conspiracy against him.
Mr. Gadfly,
I’ve seen people of all political persuasions create narratives and back-fill them with convenient facts.
Here in Nevada I’m watching the political left follow through with an old tired narrative about not taxing corporations enough and not spending enough on K-12 education. They we’ve a couple facts together, suggest because we don’t do X we reach Y result. But all they do is selectively choose their facts and conveniently ignore the facts that don’t fit the narrative.
Nevada needs to tax mining in a realistic way. (But never will, IMO.)
Mr. Gadfly,
I’ve seen people of all political persuasions create narratives and back-fill them with convenient facts.
Here in Nevada I’m watching the political left follow through with an old tired narrative about not taxing corporations enough and not spending enough on K-12 education. But all they do is selectively choose their facts and conveniently ignore the facts that don’t fit the narrative.
But you see, that’s where I disagree with you, or at least your perception of spin. I felt it was a pretty solid article. Sure, it could have been more nuanced and detailed but I thought Jonah’s point was to convey precisely that science is messy and doesn’t deliver ultimate truths. That’s a good thing in my book because it demythologizes science.
And based on your perception of his “spin”, you then lumped him in with denialists and nihilists. I think that’s doing one of the best young science writers out there a huge disservice. The former implies rejection of science while the latter implies a lack of concern for science. Anyone who reads Lehrer’s blog regularly can tell that both of those were false.
However, those two points aside I think your criticisms are excellent so I’m just going to have to agree to disagree with you on the intent and effect of Lehrer’s article.
I agree that his blog paints a different picture. I disagree about the impression that his New Yorker article creates – I have received a ton of e-mails from people who read it and thought he was being anti-science.
Perhaps his failing in this piece was not recognizing the context sufficiently – the context that I discuss above. Next time he should actively distance himself from the misperception of his point as being nihilistic, rather than making statements that are begging to be interpreted that way.
And yes – he did make good points about science, as I said – all point that skeptics have been making for years.
The whole time I was reading the article,I kept thinking,”where is he going with this?”.By the end I felt like I wasn’t sure what the ultimate message was.It seems like depending on what your initial bias was before reading it,you could easily come to rather different conclusions by the end.
I would wager that people in all 3 categories that Steve wrote about here will be referencing Lehrer’s piece to bolster their positions.
Talk about spin. The article’s subheadline is “Is there something wrong with the scientific method?”
Did Lehrer write the headline?
Whether you’re diagnosing a disease or testing ESP, you often start with high sensitivity to make sure you don’t miss anything serious, and then increase specificity to reduce false alarms. Is that a “decline effect”? Even the JREF Million Dollar Challenge has a preliminary test and a formal test.
I take it a “science denier” is anyone with an opposing viewpoint on some issue on which your side has tried to wrap itself in the mantle of scientific authority?
Tri – “I take it” can be interpreted as “my ideological bias, which I will not further examine or question.”
Since I gave a fairly operational definition of “science denial” (at least in this context, and there are many other articles on this site which further explore this concept) this is not simply a way of dismissing contrary opinions. Ironically, your reaction is dismissive.
I have the impression that there is a shift in the public perception of science compared to real science. Maybe it’s because of things like “Fringe”, which is fantasy painted as fringe science, but it seems the public only is aware of science when the subject is actually fringe science, and see as fringe science things that are actually pseudocientific, while established science passes as every day common sense.
So, when people behave like or say they are anti-science, I see it generally as more a matter of semantics than anything else: such people are rarely “anti” Gravity or anti using their cell phones or computer or driving their cars. When they say they don’t trust science, for the most part what they actually mean is they don’t trust science that is not well established yet or has not been adopted widely.
The problem lies then, it seems, with the science that is at any time making the transition from “fringe” to “established”, so to speak, and finding the best way to teach people the difference between both.
So maybe the answer is for scientists not to publicize their findings? I mean that quite seriously. If almost all new research is flawed then the responsible thing to do is not to talk about it outside of the journals. That would have prevented NASA from making itself look stupid. Transparency is harmful if it constantly misleads the public.
@William,
I will second that. Reminds me somwehwat of Sagan in “The Demon Haunted World” when the Soviet’s held a press conference to report the discovery of radio emissions from CTA-102 and the hypothesis of ET life. It became an immediate sensation in the media – and of course it was wrong.
Vergil Den
Speaking of….here is a good example of questionable science http://bit.ly/hOU2Rj
Vergil Den
Lukas, it is absolutely amazing: we still have to choose what to believe. Most people think we choose what we believe and they scratch and fight until their fingernails bleed to support this ridiculous notion.
The idea that human belief is not something we select from a menu, but something that comes to us based on the facts we see (or the faith we have), seems to be very elementary.
I choose to love to do housework, to believe it is fun. Why am I composing this in filth?
For the next ten minutes, believe the earth is completely flat, starting when I say go.
Ready …
Go!
Is the earth completely flat?
What, couldn’t do it. Too bad.
OK, one more time….
Go!!!
No, damn!