Experimental Philosophy

Knobe in the NYT

Our own Joshua Knobe has an interesting piece on theory of mind and corporations that just appeared in The New York Times entitled, "Do Corporations Have Minds?" He reports some interesting findings that address a very important moral, legal, and political issue (especially given some of the awful U.S. Supreme Court decisions from the past few years about these and related issues).  So, go check it out!

Nifty Tricks

I like nifty tricks that I can use in data exploration and analysis. Here is one that I just came across today:

Daniel Lakens used a simulation to illustrate why you should always report Welch's t-test, which does not assume equal variance, and not Student's t-test -- even when using a two-step procedure that involves Levene's test. It's a simple trick, and it actually makes things simpler, and hey, you're doing the statistically right thing!

Here is another one I came across a while back:

Chris Said argued that bar graphs with error bars, for all their faults, can be useful for eyeballing comparison of means. However, instead of using the standard 95% confidence interval for means, the eyeballing trick involves using 83.4% CI for the error bars. Now, there are more limitations to this trick, but it's still useful.

So, those are the two nifty tricks that I learned recently. What are yours?

Venues for X-Phi

I have been struggling to decide on a suitable venue for a paper I'd like to submit to a journal. On the one hand, it's a bit longer than most journals prefer (~13,000 words).  On the other hand (and more importantly), it contains the results of several studies. Other than the interdiscplinary journals like Mind and Language, Consciousness and Cognition, and Philosophical Psychology, which mainstream philosophy journals have published work by experimental philosophers? I, for one, co-authored a piece on free will in PPR several years ago and I published another piece in Analysis (from all the way back in 2004!). But I was curious what kind of luck others have had publishing their work in mainstream journals. More importantly, what kind of experiences have experimental philosophers had working with traditional journals--e.g., did you get helpful referee reports or did the editors even send out your work to referees before rejecting it straightaway?  Any and all feedback would be greatly appreciated!

Why Do Experiments?

On psychologist Simine Vazire's always-excellent blog, sometimes i'm wrong, there is an excerpt from John Doris's forthcoming book that reacts to #repligate. Doris makes many important points about how philosophers should respond to this episode in psychology, such as not relying too much on any single study, including any single replication.

However, I want to take issue with one parenthetical remark. Doris writes "(Less cynically: if scientific findings weren’t surprising, why would we need experiments and publications?)". Although this may just be a throwaway remark for Doris, I actually think it might be a somewhat common thought. The thought is that experiments get some of their value from surprisingness -- i.e. disconfirming some intuitive thought. Or, put it in the reverse direction, if people were able to reliably predict whether an experiment would confirm or disconfirm a commonsense belief, then there would be less reason for us to do such an experiment.

I don't think that's the right view about experiments. Duncan Watts's book Everything is Obvious gives an easy way to see why. As Watts points out, many of our commonsense beliefs appear in tension with one another. Absence makes the heart grow fonder. But out of sight is out of mind. So you might imagine that, if the result of an experiment went one way, it'd be framed as "Counter-intuitive finding! Absence does not make the heart grow fonder!" for publication, but if the result of the same experiment went the other way, it'd be framed as "Counter-intuitive finding! Out of sight is not out of mind!" for publication.

Given that many of our commonsense beliefs appear in tension with one another in this way, whether an experiment is counter-intuitive or not hardly has any connection to its value. Instead, as Watts points out, what is valuable about scientific experiments is that they can delineate the scope of such commonsense beliefs. And I might add, they can estimate the magnitude of the highlighted causal relationships. From a scientific perspective, it's not that interesting to find out whether "out of sight is out of mind" is true or not, but it is interesting to find out in which cases it holds and in which case it doesn't, and to what extent it holds.

I think this view of why do experiments actually coheres with some important methodological lessons that came out from #repligate. There should be less emphasis on the yes/no question of whether p is above or under 0.05. Instead, there should be more emphasis on effect sizes, which measure the magnitude of a causal relationship, including giving confidence intervals for effect sizes. Moreover, there should be more emphasis on "scope" -- covering aspects ranging from moderating factors to conditions under which the effect is expected to replicate. For example, as I've mentioned before on this blog, psychologist Dan Simons suggests that publications of studies should come with an explicit method section on "Limits in scope and generalizability". At the same time, Many Labs 2 and Many Labs 3 are now investigating, respectively, replication across sample and setting and replication across timing of experiments.

Why do experiments? The answer has nothing to do with intuitiveness, but everything to do with magnitude and scope of causal relationships that are of interest to some of our aims.

[x-posted at Experimental Philosophical Aesthetics]

How to Better X-Phi

Many readers of this blog have probably already seen the latest issue of Emotion Review, which features Nina Strohminger's review of Colin McGinn's book on disgust. There is also a terrific set of commentaries that expand on the themes prompted by Strohminger's review.

In particular, David Pizzaro's commentary is focused on the challenges of interdisciplinary research. Pizzaro rightly joins Strohminger in excoriating McGinn's data-free approach to theorizing disgust, and noting the essentiality of data to making scientific claims. Pizarro also rightly emphasizes the importance of intellectual humility in doing interdisciplinary research. Toward the end of the commentary, Pizarro writes,

Philosophers interested in empirical questions about the mind, but who have not themselves been trained in experimental methods or statistical analyses, are more likely [...] to selectively report results that support their argument [...]

I find this claim curious. Is it the case that experimental philosophers who lack formal background in experimental methods or statistical analyses are more likely to selectively report results that support their argument?

Given the recent discussions regarding replication and research practices in psychology, what can experimental philosophers do to avoid being seen by other philosophers and psychologists as p-hackers? The replication project in X-Phi is a good start. Perhaps we should be p-curving ourselves more? Perhaps we should be registering our projects on Open Science Framework?

What other measures do you think would be helpful toward a better x-phi?

Unenlightened editorial at Theoria

Sven Ove Hansson, prolific scholar and editor-in-chief of Theoria, published an opinion piece on the noun phrase "Experimental Philosophy" in the latest issue of Theoria (subscription required). He makes some provocative and, I think, pernicious claims. I'd like to list a few of them here and offer some brief commentary. But it would be really great if other people shared their thoughts too. As will be obvious, I'm mainly in disagreement with Hansson, but perhaps others think Hansson is on the right track.

(1) The field is "largely dominated by surveys," and "the use of the term 'experimental' for the collection of survey data is severely misleading." We should "reserve the term 'experiment' for those empirical observations in which some object of study is subjected to interventions (manipulations)." The implication: work in the field is "non-experimental."

Maybe others can point to some examples, but I am unaware of any recent work that reports mere surveys. All the work I read and cite in the field is experimental in exactly the sense that Hansson favors: it manipulates variables and then analyzes the data to see whether the manipulation affected the outcome.

(Incidentally, non-experimental studies can sometimes be highly informative.)

(2) Experimental philosophers study people's "spontaneous, unimproved reactions," which is inadequate because "arguably, what philosophers call intuitions are often views and standpoints" reached in "reflective equilibrium rather than a pre-reflective standpoint." To study people's reflective equilibria, we'd need to do "experimental studies comparing groups that are given access to different types of facts and arguments."

Lots of great work on this has already been done and more is on the way. Experimental philosophers have already amassed an impressive amount of data on the judgments of trained philosophical experts, including comparing them to the judgments ordinary people, comparing them to the judgments of other experts, and probing for effects of culture, personality, language, and disciplinary background, among other things. I submit that this counts as comparing groups that have "access to different types of facts and arguments." Quite often, philosophical expertise doesn't seem to make much of a difference.

(Incidentally, I'd argue that for many philosophically important categories associated with core competencies in social cognition, the average healthy adult definitely has relevant expertise. Young children often do too.)

(3) "Philosophy needs to be enriched with much more empirical data. But for the most part, others are much better qualified than philosophers to assemble that data."

The more enrichment and engagement, the better. Definitely. But, last time I checked, experimental philosophers were publishing their experimental papers not only in top philosophy journals, but also top cognitive science and experimental psychology journals. Many experimental philosophers are also either trained psychologists or cross-appointed in cognitive science. In light of these facts, I submit that lots of philosophers are highly qualified to do experimental work.

Moreover, in my experience, professional social scientists sometimes (I reiterate: sometimes) aren't all that interested in running the experiments we need to address the philosophical issues we're focused on. (Sometimes they're right to not be interested, sometimes not.) Collaboration is definitely great, but in these cases, why wait around for others when we can do it very well ourselves?

(4) "Philosophy, on its side, has access to a range of useful methodologies that are specific for our discipline. It seems to be a good idea to focus on doing that which we can do better than others rather than that which others do better than we."

Can anyone name a useful technique or mode of cognition that pretty much only philosophers rely on, and which counts as a "source of philosophical knowledge"?