Several experimental philosophers have found that philosophical intuitions of both professionals and laypeople are sensitive to the order in which the thought experiments they come from are presented. A nice summary of the latest results about this by Schwitzgebel & Cushman can be found here.
But now, not one but two new papers critical of this body of research are set to appear in Synthese (what did order effects ever do to you, Synthese?). One paper “Ordering effects, updating effects, and the specter of global skepticism” by Zach Horne & Jonathan Livengood questions the assumption that order effects indicate philosophical intuitions are unreliable. Another paper, “How not to test for philosophical expertise” by Regina Rini claims to have identified a deep methodological problem invalidating most findings about order in philosophy!
Zach Horne & Jonathan Livengood offer two major insights about order effects. The first is that not all order effects are created equal. Some order effects are pernicious. But other order effects—they classify these as “updating effects”—are not so pernicious. These kinds of effects are very close to what regularly happens when we update our beliefs during learning. The authors argue that many findings in the literature are of this latter type, and shouldn’t be taken as indications of unreliability. The second major insight involves the observation that order effects are incredibly common features of human cognition. They are so common, the authors argue, that assuming order effects are even prima facie unreliable would undermine our confidence in almost everything human beings do. Given this skeptical consequence, the authors conclude assumptions about reliability should not be made on the basis of order.
This raises an interesting question. Just what is it responsible to conclude about reliability when order effects occur in professional philosophical intuition? Perhaps many things associated with order effects are actually pretty reliable. At the same time, it’s hard to identify what new thing professional philosophers might have learned about Trolleys when taking the experiments, or if pervasiveness of effects is ultimately very conducive to expertise in giving intuitions.
Regina Rini’s paper is also based on a major insight about this research. Her insight is that professionals and non-processionals may be completing different tasks in these experiments. Philosophers are much more familiar with the cases and have probably already reached opinions about them. One hypothesis then, is that they share their previously worked out opinions as answers in controlled experiments, whereas laypeople give opinions for the first time. Rini then further hypothesizes that this difference could be important when making predictions about the results of the experiment. For example, if the process of sharing previously worked out opinions is not susceptible to cognitive bias, and if that’s what philosophers were doing all along, then we shouldn’t expect philosophers to give biased answers. But philosophers do end up doing that. Rini concludes that this unexpected result is an anomaly that cannot be satisfactorily explained and thus that findings based on this method should be rejected.
This raises another interesting question. How are people processing philosophical thought experiments and do levels of familiarity help or interfere with doing so in some way? In the present case, we don’t have a lot of evidence for how philosophers process familiar cases, and no evidence is given that processing familiar things isn’t susceptible to cognitive bias sometimes. So whether it is unexpected or not, another way to go is to uphold the data and reject one of those premises.
Comments