Experimental Philosophy

On unfinding

If a result fails to replicate, then it is not a finding but rather an unlucky aberration. This post isn’t about replication failure (or replication-failure failure). It’s about something I call “unfinding.”

The following combination is not an uncommon occurrence:

(1) a result (“R”) replicates and is a finding, but
(2) different stimuli or procedures produce a different result (“R*”).

In many cases, the combination of (1) and (2) provides evidence for a more detailed or precise interpretation of R. Of course, in any particular case, people might reasonably disagree about whether the added detail constitutes a significant advance in our understanding of the underlying issues. Still, I think that there should be a presumption in favor of welcoming and encouraging work aiming to add detail.

Unfinding goes beyond merely reinterpreting R. Instead, unfinding occurs when (1) and (2) support a more radical conclusion, namely:

(3) R is uninformative (relative to the primary research question).

Uninformative findings can be set aside. They do not constrain future research on the topic, except insofar as it should not repeat the mistake.

Here is a hypothetical example illustrating an unfinding. Someone hypothesizes that the concept of physical beauty is partly constituted by visual symmetry. In a series of studies, manipulating visual symmetry strongly affects beauty judgments (R). This is taken to support the hypothesis. Subsequent research reveals that the symmetry manipulation was systematically confounded with changes in hue or saturation, and once those factors are controlled for, symmetry does not affect beauty judgments (R*). In this case, I think it’s fair to say that the original result is not genuinely informative relative to the primary research question. As an indication of this, if subsequent research on beauty judgments failed to find an effect of visual symmetry, the authors would not be obliged to explain the apparent inconsistency with the original result.

Unfinding is a possibility for just about any result, and we should be open to it in principle. Nevertheless, I do not think that there should be a presumption in favor of welcoming or encouraging work that aims at unfinding.

Unifying Morality's Effect on Non-moral Cognition

At this point, it is pretty clear is that people’s moral judgments affect a surprisingly large number of their judgments that do not seem to be straightforwardly moral (e.g., belief, causation, doing vs. allowing, freedom, happiness, innateness, intentional action, knowledge, love, and so on) 

The sheer number of different judgments affected by morality provides some reason against continuing to search for separate explanations for each effect. Rather, it looks as though we should be searching for ways in which the various effects of morality may be unified. Of course, the sort of explanation that could unify these various effects would have to be so abstract that it could be playing a role in each of these diverse kinds of judgments.

In a new paper, we propose that the key to understanding how these effects can be unified is to begin thinking about the role of alternative possibilities. The basic idea is that our way of understanding many aspects of the world involves not only thinking about the things that actually happened but also considering other alternative ways that things could have happened but didn’t. Along these lines, our proposal has two key components:

(1) Morality affects which alternative possibilities are relevant.

(2) Each of the ordinary judgments affected by morality involve the representation of alternative possibilities in some way or another.

To test this unifying framework, we took the original materials from four previous studies which showed that agents who acted immorally (vs. neutrally) were more thought to have acted freely, caused the outcome that occurred, did something rather than allowed it to happen, and acted more intentionally. Using the original materials, we both replicated the four original effects, and also found that changes in the morality of the agent’s action impacted participants’ judgments of the relevance of alternative possibilities. Critically though, we also found that participants’ judgments of the relevance of alternatives mediated the effect of morality on judgments of freedom, causation, doing vs. allowing, and intentional action.

 

We then sought to further confirm this general framework by manipulating rather than measuring changes in the relevance of alternatives.

 

What Has Experimental Philosophy Discovered about Demographic Effects?

In a recent post, I discussed a new study showing that people's intuitions about Gettier cases are remarkably invariant across cultures. This post generated a whole lot of discussion on social media, and although most of this discussion was extremely helpful and well-informed, I worry that some of it betrays a confusion about what experimental philosophy is all about. 

Some commenters seem to assume that it is somehow built into the mission of experimental philosophy as a field that part of its aim is to show that intuitions vary from one demographic group to the next and that appeals to intuitions are therefore methodologically suspect. As anyone who actually works in experimental philosophy will know, this assumption is wildly incorrect. The mission of experimental philosophy as a field is not argue for any one particular view but rather to systematically study these questions and find the truth about them, whatever that truth might be.

Consider again the particular case of research on the relationship between philosophical intuitions and demographic factors (ethnicity, gender, age, etc.). It is true that some experimental philosophy studies have suggested that people's intuitions vary with demographic factors, but there have also been a very large number of studies exploring the ways in which people's intuitions do not vary with these factors. In fact, my sense is that the majority of experimental philosophy studies on this topic actually come down on this second side, exploring ways in which people's intuitions are surprisingly invariant across demographic variables that one would have thought would make a difference.

To get a better sense for what has been going on in the field thus far, let's try to work together to make a list of the different studies on this topic and what they have shown.

 

A Feminist Critique of Experimental Philosophy

In the first issue of the recently launched on-line open-access journal Feminist Philosophy Quarterly, there is a piece by Gaile Pohlhaus Jr. entitled, "Different Voices, Perfect Storms, and Asking Grandma What She Thinks: Situating Experimental Philosophy in Relation to Feminist Philosophy." It has made its way onto my never-ending reading list!

Here is the abstract:

At first glance it might appear that experimental philosophers and feminist philosophers would make good allies. Nonetheless, experimental philosophy has received criticism from feminist fronts, both for its methodology and for some of its guiding assumptions. Adding to this critical literature, I raise questions concerning the ways in which “differences” in intuitions are employed in experimental philosophy. Specifically, I distinguish between two ways in which differences in intuitions might play a role in philosophical practice, one which puts an end to philosophical conversation and the other which provides impetus for beginning one. Insofar as experimental philosophers are engaged in deploying “differences” in intuitions in the former rather than the latter sense, I argue that their approach is antithetical to feminist projects. Moreover, this is even (and perhaps especially) the case when experimental philosophers deploy “differences” in intuitions along lines of gender.

While you're at it, check out some of the other interesting articles in the volume as well. The more journals like this, the better.

Does X-Phi need to assume Centrality?

In my new paper in Philosophical Studies, I explore recent arguments by Cappelen, Williamson, and Deutsch that intuitions are not used as evidence in philosophy.  I argue that a rejection of 'Centrality' doesn't undermine the importance of experimental criticisms of philosophical method.

Mturk Now More Expensive

As many readers have probably heard, Amazon's Mechanical Turk will soon (July 22, 2015) be charging substantially more for its services. Before, Mturk took 10% of what researchers ("requesters") paid workers. Now it will be effectively 40% (in the rare case that you need fewer than 10 distinct participants, then it's only 20%).

Naturally, some researchers are upset, as documented at The Chronicle of Higher Education ("Researchers Complain About Changes in Amazon Tool Used for Surveys").

Two quick downsides that stand out to me:

(1) Running even a small x-phi study on Mturk will more likely require grant money, which is scarce in the humanities.

(2) An increase in operating costs will probably motivate researchers to still get the same mileage out of their funds. This makes it even less likely that workers will get paid a living wage, which is an ongoing ethical issue with use of the site.

Any reactions from the x-phi community?

(Note: If interested, this blog recently featured an excellent post on the status of Mturk as a research tool.)

Knobe in the NYT

Our own Joshua Knobe has an interesting piece on theory of mind and corporations that just appeared in The New York Times entitled, "Do Corporations Have Minds?" He reports some interesting findings that address a very important moral, legal, and political issue (especially given some of the awful U.S. Supreme Court decisions from the past few years about these and related issues).  So, go check it out!

The State of Reproducibility in Experimental Philosophy

Over at Feminist Philosophers, Dan Hicks casually mentions in a comment:

I haven’t had too many interactions with folks interested in experimental philosophy, but based on those interactions it seems like (1) x-phi folks are modeling their methods explicitly on these kinds of studies in psychology, while (2) not being aware of or responsive to the epistemic crisis surrounding these methods in psychology. So I would encourage x-phi folks to check out the links above.

Despite some initial defensiveness, I've come to think that Hicks's remarks are actually quite reasonable for those philosophers who haven't really kept up with the nitty-gritty happenings in experimental philosophy. Having heard about the replicability crisis in psychology, it is easy to assume that experimental philosophy is susceptible to the same problems. Having not kept up with the nitty-gritty happenings in experimental philosophy, it is easy to assume that experimental philosophers are unaware of or unresponsive to these issues.

The point of this post is to dispel those assumptions.

To be clear, my aim in writing this post isn't really to pick on a casual comment, but to give a more general state of the art on what experimental philosophy, as a community, is doing and has been doing about methodological and epistemological issues related to reproducible science.

* * *

To start thinking about issues with reproducibility, we can go back to the context of Hicks's comment: a "sexy" result from a low-powered social psychology study that says you can reduce implicit bias in your sleep. The replicability crisis in psychology has, in large part, to do with with replicating findings of a similar sort. As such, having learned about the replicability crisis, a philosopher is prima facie reasonable to cast a skeptical eye toward experimental philosophy findings.

The skeptical eye, I take it, has to do with a companions-in-guilt charge. Experimental philosophy often borrows methods from (social) psychology; since there's a crisis in psychology, would it be such a surprise if experimental philosophy doesn't suffer from the same methodological and epistemological issues?

There is something to the companions-in-guilt charge, but we should clarify who the companions really are. It's not just psychology that suffers from a replicability crisis. It's also cancer research, pharmaceutics, political science, behavioral economics, etc. In other words, there is a replicability crisis in science. So, philosophers should worry about experimental philosophy... insofar as and as much as they are worried about science.

* * *

 Is there reason to be more worried about experimental philosophy compared to psychology and other sciences? Not only do I think the answer to that question is 'no', but I think we in fact have reasons to be relatively optimistic about experimental philosophy.

The experimental philosophy community has been aware of the issues that underlie the replicability crises even before their recent publicity. For a long time (in a young research program), Joshua Knobe (and later Christian Mott) has maintained the experimental philosophy replication page, which documents experimental philosophy findings that have (and have not) replicated well. Looking at the replication page, one can see that the majority of effects in experimental philosophy have, in fact, been well-replicated. (The big exception is the cluster of demographic effects.) So, not only are experimental philosophers aware of methodological and epistemological issues with replicability, they have already taken some steps to address it.

The replication page is one exemplar of a broader pattern. The community, in general, is very welcoming to cumulative research practices, which encourage internal corrections and challenges. For example, on this very blog there was an extended discussion about the replicability of an age effect. For example, Chandra Sripada and Sara Konrath shared their data so that David Rose, Jonathan Livengood, Justin Sytsma, and Edouard Machery can re-analyze it to challenge Sripada and Konrath's interpretation. For example, Adam Feltz and Florian Cova have systematically examined the moral responsibility and free will literature via a meta-analysis.

I'll let you in on a little secret. In the earlier days of experimental philosophy (and to a lesser extent even today), many people ran their online studies using Knobe's Qualtrics account. So everyone can in principle see what everyone else is working on, and even download the data if they wanted. In my view, the general culture of transparency discourages outright frauds like data manipulation or even faking data.

In addition to a general culture of transparent and cumulative research, experimental philosophers may also be in a better position than, say, psychologists because, well, they're philosophers. And philosophy has a long tradition of thinking about the relationships between experiments, statistical inference, theory confirmation, etc. Some experimental philosophers are also philosophers of science who are attentive to such issues. For example, Edouard Machery has written about the interpretation of null results. But even the experimental philosophers who don't also specialize in philosophy of science will have some exposure to these issues. And, based on the attendance at the Preconference Workshop on Replication in the Sciences at SPP, experimental philosophers are very keen to learn even more about these issues from both philosophers of statistics like Deborah Mayo, as well as methods experts from diverse cognate fields.

* * *

None of this is to say that current practices of experimental philosophy are anywhere near perfect. There is much more we can all do to do better: post data on repositories, run better powered studies, check with statistical and methodological consultants, keep up with latest best practices, etc. [I also have some thoughts about how to make the experimental philosophy replication page even more useful, but I'll save that for another occasion.] But we can acknowledge the need for improvements in the future, while also acknowledge the efforts for improvements in the past. We can recognize how far experimental philosophy still has to go, with respect to the methodological and epistemological issues associated with the replicability crises, while also recognize how far it has come.

[Much of this post is based on a talk at the 2014 Experimental Philosophy UK Workshop. I thank the organizers for giving me the occasion to think more systematically about these issues.]

Blackwell Companion to Experimental Philosophy

I am happy to announce a forthcoming new volume from the Blackwell Companions to Philosophy Series: A Companion to Experimental Philosophy. It is edited by Justin Sytsma and myself and features over forty chapters from leading philosophers, psychologists and cognitive scientists from several countries across three continents. These chapters detail the application of empirical methods to philosophical questions across major sub-disciplines including the philosophy of action, moral and political philosophy, philosophy of mind, epistemology, philosophy of language, metaphysics, logic, and metaphilosophy.

The general introduction to the volume is available online here, offering a summary of the goals of the volume and a brief introduction to experimental philosophy for a general audience. The table of contents and list of contributors can be found below:

Nifty Tricks

I like nifty tricks that I can use in data exploration and analysis. Here is one that I just came across today:

Daniel Lakens used a simulation to illustrate why you should always report Welch's t-test, which does not assume equal variance, and not Student's t-test -- even when using a two-step procedure that involves Levene's test. It's a simple trick, and it actually makes things simpler, and hey, you're doing the statistically right thing!

Here is another one I came across a while back:

Chris Said argued that bar graphs with error bars, for all their faults, can be useful for eyeballing comparison of means. However, instead of using the standard 95% confidence interval for means, the eyeballing trick involves using 83.4% CI for the error bars. Now, there are more limitations to this trick, but it's still useful.

So, those are the two nifty tricks that I learned recently. What are yours?