Why Do Experiments?

On psychologist Simine Vazire's always-excellent blog, sometimes i'm wrong, there is an excerpt from John Doris's forthcoming book that reacts to #repligate. Doris makes many important points about how philosophers should respond to this episode in psychology, such as not relying too much on any single study, including any single replication.

However, I want to take issue with one parenthetical remark. Doris writes "(Less cynically: if scientific findings weren’t surprising, why would we need experiments and publications?)". Although this may just be a throwaway remark for Doris, I actually think it might be a somewhat common thought. The thought is that experiments get some of their value from surprisingness -- i.e. disconfirming some intuitive thought. Or, put it in the reverse direction, if people were able to reliably predict whether an experiment would confirm or disconfirm a commonsense belief, then there would be less reason for us to do such an experiment.

I don't think that's the right view about experiments. Duncan Watts's book Everything is Obvious gives an easy way to see why. As Watts points out, many of our commonsense beliefs appear in tension with one another. Absence makes the heart grow fonder. But out of sight is out of mind. So you might imagine that, if the result of an experiment went one way, it'd be framed as "Counter-intuitive finding! Absence does not make the heart grow fonder!" for publication, but if the result of the same experiment went the other way, it'd be framed as "Counter-intuitive finding! Out of sight is not out of mind!" for publication.

Given that many of our commonsense beliefs appear in tension with one another in this way, whether an experiment is counter-intuitive or not hardly has any connection to its value. Instead, as Watts points out, what is valuable about scientific experiments is that they can delineate the scope of such commonsense beliefs. And I might add, they can estimate the magnitude of the highlighted causal relationships. From a scientific perspective, it's not that interesting to find out whether "out of sight is out of mind" is true or not, but it is interesting to find out in which cases it holds and in which case it doesn't, and to what extent it holds.

I think this view of why do experiments actually coheres with some important methodological lessons that came out from #repligate. There should be less emphasis on the yes/no question of whether p is above or under 0.05. Instead, there should be more emphasis on effect sizes, which measure the magnitude of a causal relationship, including giving confidence intervals for effect sizes. Moreover, there should be more emphasis on "scope" -- covering aspects ranging from moderating factors to conditions under which the effect is expected to replicate. For example, as I've mentioned before on this blog, psychologist Dan Simons suggests that publications of studies should come with an explicit method section on "Limits in scope and generalizability". At the same time, Many Labs 2 and Many Labs 3 are now investigating, respectively, replication across sample and setting and replication across timing of experiments.

Why do experiments? The answer has nothing to do with intuitiveness, but everything to do with magnitude and scope of causal relationships that are of interest to some of our aims.

[x-posted at Experimental Philosophical Aesthetics]

Comments

[Jason D'Cruz]

Interesting post!

I wonder whether we can ever really purge science, and especially social psychology, of the bias in favor of counter-intuitiveness. The reason why many people are interested in social psychology is that they are interested to learn whether commonly held assumptions about human behavior and interaction are true. We might doubt that they are. But to learn that they are true after all is not very exciting. It doesn't feel like learning. (Even if it is).

Your point about our commonsense beliefs being in tension with each other is an apt reply to this. But I wonder whether our commonsense psychological beliefs really are like this. Consider again the platitudes you mentioned: "Absence makes the heart grow stronger" and "Out of sight, out of mind". I suspect that we already have commonsense beliefs about the scope of these beliefs. For example, if you know the date when your beloved will return, and that date is not too far off, and you think about your beloved a lot, then your beloved's absence will increase your desire. On the other hand, if you don't care very much about the absent person, and you are tempted to do something that they would frown upon, you are likely to ignore their preferences when they are gone. etc, etc, etc...

Now, if these commonsense beliefs about the scope of folk psychological platitudes were shown to be false by psychological science, that would be interesting. But the interest would like in their counter-intuitiveness, no?

[Josh May]

Nice post about an excellent excerpt! I totally agree with your points. I'll add this: Sometimes it's hard to tell what's intuitive or common sense anyhow. And sometimes what really is intuitive or common sense can still be in some sense surprising and important to prove empirically.

Here's an example. I'm teaching Hume in an Ethical Theory class right now and going over his views about sympathy. From the armchair, Hume (and others) have realized that sympathy/empathy is partial to those we care more about, are closer to, etc. Yet Hume also thought we can sometimes be genuinely altruistic. I think this is all really common sense, but we often forget these facts or don't dwell on them. The later experimental findings supporting Hume's ideas (see e.g. C. Daniel Batson's awesome work), which Hume didn't have access to, are in some sense surprising. Even if they're not surprising to some, they're important at least because some theorists deny the apparent truisms (e.g. egoists). At any rate, such findings are important to uncover scientifically to make sure we have an accurate picture of the mind, regardless of intuitiveness, as you say.

[Shen-yi Liao]

@ Jason

Thanks for the response. It's really hard for me to even make armchair conjectures about my own mental states, but I would guess my confidence in the scope claims are much lower than in the first-order truisms. So, even if I learn something "intuitive" of that sort, I could still feel like I've learned something because I get a big increase in confidence. Similarly for magnitude. But, that's just a guess.

@ Josh

Thanks for the reference! I agree, it's really hard to tell what's intuitive or not. We're fairly clever creatures, and we can assimilate evidence really quickly so that what used to be counter-intuitive now seems intuitive. Indeed, that's one thought behind the full title of Duncan Watts's book: Everything is Obvious (Once You Know the Answer).