Experimental Philosophy

Actually, women are MORE likely to share their (mostly-male) professors' intuitions...

...at least for the 9 questions I studied for my 2013 paper, and for the 3 of those that Buckwaler and Stitch included in their widely-discussed paper on the subject. Plus: why has philosophy made so much less progress toward gender equality than the STEM disciplines have over the past two decades? I address these issues in my new article in Hypatia.

Negative Public Perceptions of Philosophical Research

Philosophy is not always perceived as well as philosophers would like. There are public discussions of whether philosophy has any value at all (The New York Times or The Chronicle of Higher Education). Members of our own draft manifestos criticizing philosophy's contributions to the academy (see Glymour’s “Manifesto” or Thagard’s “Eleven Dogmas of Analytic Philosophy”). Presidential candidates quip we need less philosophers.

In the latest incident, my personal childhood hero, Bill Nye got skeptical with BigThink about how far philosophical methods can take you in life, compared to say, scientific ones:

  

What contributes to negative opinions about philosophical research?

A Response to Critiques of Cushman's and My Work on Philosophers' Susceptibility to Order Effects

The order in which moral dilemmas are presented matters to people's judgments and can substantially influence later judgments about abstract moral principles. This is true even among professional ethicists with PhD's in philosophy. In 2012 and 2015, Fiery Cushman and I published empirical evidence supporting these claims. We invite a metaphilosophical conclusion: If even professional philosophers' expert judgments are easily swayed by order of presentation, then such judgments might not be stable enough to serve as secure grounds for philosophical theorizing.

Synthese has recently published two critiques of the literature on order effects in philosophy, which address Fiery's and my work (HT Wesley Buckwalter). Both critiques make valuable points. However both also admit of some clear replies.

To fix ideas, consider two versions of the famous Trolley Problem:

Push: A runaway boxcar is headed toward five people it will kill if nothing is done. Jane can stop the boxcar by pushing a hiker with a heavy backpack in front of the boxcar, killing him but saving the five.

Switch: A runaway boxcar is headed toward five people it will kill if nothing is done. Vicki can stop the boxcar by flipping a switch to divert it to a sidetrack where it will kill one person instead of the five.

Fiery and I presented Push-type and Switch-type scenarios (fleshed with a bit more detail) to professional philosophers and two comparison groups of non-philosophers. We found that when professional philosophers saw a Push-type scenario before a Switch-type scenario, 73% rated the two scenarios equivalently on a 7-point scale. Then later in the questionnaire when asked about the Doctrine of the Double Effect -- a moral principle often interpreted implying that Push-type cases are morally worse than Switch-type cases -- only a minority, 46%, endorsed that principle. In contrast, among philosophers who saw Switch before Push only 54% rated the two scenarios equivalently, and then later a majority, 62%, endorsed the Doctrine of the Double Effect. Endorsement of the principle thus seemed to shift, post-hoc, to rationalize philosophers' order-manipulated judgments about the scenarios.

We found similar effects for Action-Omission, Moral Luck, and "Asian disease" type cases (though not consistently for every measure across the board). Philosophers with PhDs and self-reported competence or specialization in ethics showed no smaller effects than other philosophers or than comparison groups of non-philosophers -- and in fact trended slightly (non-significantly) toward showing larger order effects.

In general, we found pretty substantial effect sizes, suggesting substantial instability of judgment even in philosophical respondents' areas of expertise. Hence the metaphilosophical worry.

----------------------------------------------

Critique by Zachary Horne and Jonathan Livengood.

Horne and Livengood make three main points about the literature on order effects in philosophy:

(A.) First, they helpfully distinguish between what they call "updating effects" and "genuine ordering effects". Genuine ordering effects, in their terminology, are effects measured only after all the stimuli have been presented. "Updating effects" are measures taken along the way, and might well reflect participants' learning. There is of course nothing irrational in judging Scenario B differently as a result of seeing Scenario A because one learned something by seeing Scenario A. Most philosophical research on order effects, they note, takes the measures along the way -- and thus might be measuring learning rather than true order effects.

(B.) Second, they point out that perceptual judgments also show order effects. Thus, if we are to reject any type of evidence that shows order effects, then we must reject perceptual evidence too, which would lead to radical skepticism.

(C.) Third, they point out that order can sometimes reasonably make a difference to the evaluation of evidence. For example, a smile followed by a frown, on the same person's face, is a different type of evidence than a frown followed by a smile.

On (A): I find the labels tendentious (since if we know there isn't learning-type updating going on, what we might want to call "genuine order effects" can plausibly be measured mid-stream), however it probably is correct that most studies do not sufficiently rule out the possibility of learning or updating in the course of the experiment, if they have novice participants and take the measurements after each scenario rather than after both scenarios. However, since our participants were experts, we think it unlikely that a significant number learned anything in the process of our brief experiment that would rationally justify shifting their judgment about the equivalency or non-equivalency of Push and Switch. And as Horne and Livengood note, our measure of endorsement of the Doctrine of the Double Effect is a measurement of a "genuine ordering effect" even by their own lights.

On (B): Yes, of course it would be silly to reject all means of learning that are subject to any order effects! The epistemic sting, as they note, depends not on the mere existence of an order effect in one case, but on how large and how prevalent the order effects are. This is an open empirical question. But the limited empirical evidence that exists suggests that order effects are substantial and prevalent in moral dilemma cases. So far, we have found order effects in all of the scenario types we've tried, with about a 10-20% shift in opinion on the moral equivalency of our scenario pairs and in preference for the risky option in the "Asian disease" cases.

On (C): It's interesting to consider cases in which earlier evidence rightly colors our reaction to later evidence, but trolley problems presented to disciplinary experts seems a different kind of case.

Finally, Horne and Livengood suggest that exposure to a pair of dilemmas in our study is unlikely to have a long-lasting impact on professional philosophers' beliefs. I agree. They continue, "But if there is no long-lasting impact, then we think the effect is unlikely to matter to actual philosophical practice outside of the laboratory" (p. 17). I don't think this follows. Fiery's and my view is not that philosophers' opinions are permanently influenced by the order in which the scenarios are presented on any single occasion, but rather that their opinions are unstable -- possibly influenced one direction on one occasion, in another direction on another occasion. This instability is what drives the metaphilosophical worry.

----------------------------------------------

Critique by Regina Rini:

Rini looks only at our 2012 study. (Our 2015 study wasn't published until after her paper was in press.) She finds it plausible that if professional philosophers were already familiar with these cases they would not exhibit order effects of the sort Fiery and I find. She suggests that perhaps respondents were not previously familiar with the cases -- or at least not familiar in the right sort of way. She calls this the "familiarity problem" and offers four possible explanations:

(1.) The respondents were not really experts. She wonders if our participants, recruited through the internet, really had the degrees they claimed to have.

(2.) The respondents didn't carefully attend to our scenarios. Maybe they breezed through them so quickly that they failed to notice relevant features.

(3.) The respondents might not have familiar responses to these types of scenarios. Perhaps they have so far refrained from forming judgments on such cases and principles.

(4.) The respondents might not have diachronically stable familiar responses. This is the explanation Fiery and I favor. However, Rini helpfully points out that as long as philosophers are aware that their responses are not diachronically stable, the metaphilosophical threat is reduced: Presumably philosophers who are aware that their responses are not stable would be reluctant to ground their theorizing on those responses.

On (1): I am not aware of a general problem in the survey literature of respondents' frequently misreporting their educational status -- though certainly a bit of misreporting is possible. One specific piece of evidence against this possibility in our own study is that we recruited philosophers mostly by asking department chairs to forward a recruitment email to faculty and graduate students in their departments. Most of our "philosopher" participants took the survey within just a few days of these emails.

On (2): The median response time on the first scenario was 40 seconds, on the second scenario was 34 seconds. While these are not huge response times, if you stop to count out 34 seconds now, you'll probably notice that it's a reasonable amount of time for a thoughtful response to a brief scenario.

On (3) and (4): These are potentially quite serious issues, and in fact our follow-up study in 2015 was designed specifically to address them, after we saw an early version of Rini's critique. In our 2015 study we specifically asked participants if they were previously familiar with the scenarios. We also asked whether they regarded themselves as "having had a stable opinion" about the issues before participating in the experiment, and whether they regarded themselves as experts on those very issues. We also added a "reflection" condition to help address concern (2). In the reflection condition we asked participants to reflect carefully before responding and enforced a minimum 15-second delay between when participants reported having finished reading the scenario and when their response options appeared.

We did not find that self-reported familiarity or stability reduced the size of order effects in two different types of scenario pairs (trolley problems and risky-choice "Asian disease"-type problems), nor did we find reduced order effects in the reflection condition compared to a normal control condition without special instructions to reflect.

For example, percentage rating the Push and Switch scenarios equivalently:

[click to enlarge]

Thus, I am inclined to think that Rini's fourth suggestion is the most plausible -- that participants do not have diachronically stable familiar responses, despite high levels of expertise. But since those who report having stable responses were no less subject to order effects than were those who reported not having stable responses, self-knowledge of stability appears to be largely absent. Despite Rini's interesting suggestion that instability is metaphilosophically non-threatening if people are aware of it, Fiery's and my results suggest that we should not hasten to that comfort.

----------------------------------------------

Both Horne and Livengood and Rini emphasize that we only have very limited evidence about order effects on professional philosophers' judgments. I agree! Fiery's and my two studies are hardly decisive. Convergent evidence from several different labs would be necessary before drawing any confident conclusions, especially if those conclusions are at variance with what one feels one knows from personal experience. Rini also makes positive suggestions for follow-up experimental work that might be done, which I am inclined to support. Both critiques raise important methodological concerns that ought to help shape and direct future work on this topic.

[Cross-posted at the Splintered Mind. For discussion/comments, please go there.]

Two papers critical of order effect research at Synthese

Several experimental philosophers have found that philosophical intuitions of both professionals and laypeople are sensitive to the order in which the thought experiments they come from are presented. A nice summary of the latest results about this by Schwitzgebel & Cushman can be found here.

But now, not one but two new papers critical of this body of research are set to appear in Synthese (what did order effects ever do to you, Synthese?). One paper “Ordering effects, updating effects, and the specter of global skepticism” by Zach Horne & Jonathan Livengood questions the assumption that order effects indicate philosophical intuitions are unreliable. Another paper, “How not to test for philosophical expertise” by Regina Rini claims to have identified a deep methodological problem invalidating most findings about order in philosophy!

Intuitive Resistance is the Mind-Killer

In a recent McSweeney’s article, Kyle York proposes some interesting takes on familiar ethical dilemmas with the potential to help reveal something truly fundamental about moral judgment. For instance, pull or no pull: 

“There’s an out of control trolley speeding towards four workers. You have the ability to pull off your head and turn it into a Chinese lantern. Your head floats into the sky until it takes the place of the sun. You look down upon the planet. It is as small as the eye of a moth. The moth flies away.”

Clearly a particular way of doing philosophy and moral psychology is being mocked. Philosophers and psychologists often rely on unrealistic thought experiments and experimental stimuli when conducting their research. To be fair, they do tend to avoid the surreal, but this does raise an interesting question. How worried should we be that deep down, we don’t actually believe the things we are asked to imagine as we make philosophically and psychologically relevant judgments about them?

New Study: No Difference in Gettier Intuition Across Cultures

Within the more metaphilosophically-oriented literature on experimental philosophy, there has been a great deal of discussion of the philosophical implications of cross-cultural differences in intuitions about Gettier cases. This work has been extremely impressive from a purely philosophical perspective, but at times, I worry that it has not been sufficiently closely connected to the actual empirical work in this area. In particular, much of it starts off from the assumption that people of different cultures differ in their intuitions about Gettier cases, but it turns out that the majority of the empirical studies actually find  that Gettier intuitions do not depend on culture in this way (see here, here and here). So it sometimes seems that people are investigating the philosophical implications of an effect that doesn't actually exist. 

Happily, Noûs has just published a truly amazing study on this topic by a team of experimental philosophers (Machery, Stich, Rose, Chatterjee, Karasawa, Struchiner, Sirker, Usui & Hashimoto), and I think this new study gives us a much better understanding of the relevant empirical facts. The researchers presented two different Gettier cases to participants in the United States, Brazil, India and Japan, yielding a total sample size of 521 participants. The study is extraordinarily impressive from a methodological perspective and very much worth reading in full, but the basic result can be expressed pretty simply in the following figure:

Overall, the study finds a robust tendency, found across all four cultures, to conclude that people do not have knowledge in Gettier cases.

Of course, this finding does not mean that philosophers were mistaken to think that there was something of deep metaphilosophical importance about looking at Gettier intuitions in different cultures. On the contrary, the result obtained here is a truly fascinating one, which surely has rich metaphilosophical implications. The key point is just that the metaphilosophical question we need to be asking is the opposite one from the one people have been discussing thus far. The question worth asking is not 'What are the metaphilosophical implications of cross-cultural differences in Gettier intuitions?' but rather 'What are the metaphilosophical implications of the extraordinary cross-cultural similarity in Gettier intuitions?' This latter question has not yet been sufficiently explored, but it opens up a whole new range of exciting issues that I hope philosophers will begin exploring over these next few years.

[The full paper is available to subscribers at Noûs, but please do feel free to write in with comments even if you have not yet read the paper itself.]

A Feminist Critique of Experimental Philosophy

In the first issue of the recently launched on-line open-access journal Feminist Philosophy Quarterly, there is a piece by Gaile Pohlhaus Jr. entitled, "Different Voices, Perfect Storms, and Asking Grandma What She Thinks: Situating Experimental Philosophy in Relation to Feminist Philosophy." It has made its way onto my never-ending reading list!

Here is the abstract:

At first glance it might appear that experimental philosophers and feminist philosophers would make good allies. Nonetheless, experimental philosophy has received criticism from feminist fronts, both for its methodology and for some of its guiding assumptions. Adding to this critical literature, I raise questions concerning the ways in which “differences” in intuitions are employed in experimental philosophy. Specifically, I distinguish between two ways in which differences in intuitions might play a role in philosophical practice, one which puts an end to philosophical conversation and the other which provides impetus for beginning one. Insofar as experimental philosophers are engaged in deploying “differences” in intuitions in the former rather than the latter sense, I argue that their approach is antithetical to feminist projects. Moreover, this is even (and perhaps especially) the case when experimental philosophers deploy “differences” in intuitions along lines of gender.

While you're at it, check out some of the other interesting articles in the volume as well. The more journals like this, the better.

Does X-Phi need to assume Centrality?

In my new paper in Philosophical Studies, I explore recent arguments by Cappelen, Williamson, and Deutsch that intuitions are not used as evidence in philosophy.  I argue that a rejection of 'Centrality' doesn't undermine the importance of experimental criticisms of philosophical method.

Blackwell Companion to Experimental Philosophy

I am happy to announce a forthcoming new volume from the Blackwell Companions to Philosophy Series: A Companion to Experimental Philosophy. It is edited by Justin Sytsma and myself and features over forty chapters from leading philosophers, psychologists and cognitive scientists from several countries across three continents. These chapters detail the application of empirical methods to philosophical questions across major sub-disciplines including the philosophy of action, moral and political philosophy, philosophy of mind, epistemology, philosophy of language, metaphysics, logic, and metaphilosophy.

The general introduction to the volume is available online here, offering a summary of the goals of the volume and a brief introduction to experimental philosophy for a general audience. The table of contents and list of contributors can be found below:

Philosophers, Order Effects, and Framing Effects

Fiery Cushman and I have a new paper in draft, exploring the question of whether professional philosophers' judgments about moral dilemmas are less influenced than non-philosophers' by factors such as order of presentation and phrasing differences.

We recruited hundreds of academic participants with graduate degrees in philosophy and a comparison group of academics with graduate degrees in other fields. We gave them two "trolley problems" and two "Asian disease"-type framing effect cases.

We presented the trolley problems either with a Switch case first (the protagonist saves five people by Switching a runaway trolley onto a side track where it kills one), followed by a Push or Drop case (saving five by Pushing one person into the trolley's path or by Dropping him into its path), or Push/Drop first, followed by Switch. For each scenario, participants rated the protagonists' choice to kill the one person to save the five others, using a 7-point scale from "extremely morally good" to "extremely morally bad".

Our previous research suggests that non-philosophers are much more likely to judge the Push case and the Switch case equivalently when Push is presented first than when Switch is presented first. On some views of philosophical expertise, philosophers' judgments about these cases should be less dependent on order of presentation than are non-philosophers' judgments. In other words, philosophers, due to their familiarity with scenarios of this type and their expertise in applying moral principles to them, should have more stable opinions, less influenced by order of presentation. We wanted to see if philosophers with prior familiarity with the cases, or self-reported expertise in the area, or self-reported stability of opinion, would show smaller order effects. We also wanted to see if we could reduce order effects by encouraging participants to reflect carefully on different versions of the scenario and different ways of phrasing the scenarios and enforcing a delay before responding.

We were unable to find any level of expertise at which the order effects were detectably reduced. Nor did adding a reflection condition appear to reduce the order effects. This figure shows the rates at which Switch was rated as morally equivalent to either Drop or Push on the 7-point scale:

[click to enlarge]

The order effect, as indicated by the differences in height between the black and the gray bars, is basically the same for philosophers and the non-philosophers, even in the "reflection" condition.