Blog Coordinator

X-Phi Grad Programs

« On the Horizon at Brains | Main | The Lesson of Bypassing »



Feed You can follow this conversation by subscribing to the comment feed for this post.

Brad Cokelet

Hi Josh,


Why not stagger questions:

(a) If you were forced to choose would you say this is morally permissible or not?
(b) How confident would you be in your answer here (1 - 10)?

You could start with a sample to prime the subjects to make use of the framework.

(2) I think you might do ok sticking with permissible as I am not seeing how Hurka's stuff will ground any claims about that being gradable (if it is wrong at all, it is not permissible, right?). Similarly, I am not sure his stuff on wrongness will extend to obligation in a straightforward way.

(3) Last, I noticed one of the quoted questions was "should Josh.." I think people should be careful to distinguish should and ought claims from obligation/permission

Brad Cokelet

Or you could use a scale with thicker terms coupled with the epistemic question:

(1) This act is...[scale]

1 morally horrific/evil
2 morally impermissible
3 morally indecent/regrettable
4 morally indifferent/permissible
5 morally commendable
6 morally heroic/saintly

(2) How confident are you in your answer?


Josh May

Good points, Brad. And thanks for the feedback!

On (1): I like your initial idea, and I believe some have pursued it, although perhaps not in moral judgment research.

I'm not sure about your second proposal in your second comment, though. Your 1-6 scale doesn't clearly have equal intervals between steps, in which case it isn't a scale variable. It would have to be treated as nominal/categorical, which would be much less useful data with so many categories.

On (2): I see, you're thinking Hurka is only talking about wrongness/rightness and that this won't transfer over to permissibility/impermissibility anyhow. I guess I was just thinking that Hurka's discussion of wrongness/rightness lead me to think about whether deontic categories do come in degrees. So it's not Hurka's theory in particular that worries me (especially given how he qualifies it), but rather just the general issue his post raises.

On (3): Yes, this is somewhat dissatisfying about Zimmerman's study. Still, I think he's okay as far as targeting Mikhail. Mikhail's theory presumably predicts that people will say the agent shouldn't flip the switch in Loop, yet they did.

(Maybe Mikhail needn't predict that people will say the agent should flip the switch in Man in Front, since "permissible" doesn't necessarily entail "should." But the problem Zimmerman is raising for Mikhail is primarily that he didn't replicate the low permissibility judgments in Loop. Zimmerman's results for his version of Man in Front pretty closely match Mikhail's.)

John Turri

Hi Josh,

Great question! I've experimented with a lot of different ways to probe for cognitive evaluations. I like the "definitely yes" --> "definitely no" approach, which one might think of as a "first-order" scale for collecting responses, which asks people to judge X, as opposed to the typical "second-order" or "meta" scale one sees with Likerts that ask people to report on their view of X.

One might suspect – as I once did – that the difference between the 1st and 2nd order sclae could matter a great deal. However, in my experience, results observed with the 1st-order scale never differed significantly from what I observed with analogous 2nd-order scales. (I've not written anything on this and I'm going from memory here, so take it for what it's worth.)

In the end, I tend to favor the rather unoriginal view that when we think we've discovered something interesting about people's opinions on a certain matter, probing in different ways to see if it holds up is worthwhile (though not necessarily obligatory). If it does, great. If it doesn't, that's important information too.

By the way, I think 'neutral' and 'unclear' are perfectly respectable midpoints that don't move one off of the spectrum, and which are more natural than the 'neither/nor' or 'in between' anchors.


My first reaction was like John Turri's. I wonder whether there is any large difference between the various ways of asking people for moral judgments. Should the difference be filed away with question order and font size, or are they really getting at distinct judgments people tend to make?

If you could find that two sets of questions, e.g. judging confidence that P is wrong vs. judging wrongness of P, were poorly correlated or can come apart, this could be very interesting. It would also be mildly interesting to know that they do not come apart, as 'certainty of it being wrong' and 'how wrong it is' might be psychologically equivalent in the questionnaires.

Josh May

Thanks for the comments, John and Taylor! You both make excellent points.

Taylor, I like your idea that one could empirically test some of these issues. I've been thinking about that myself. But I'm not entirely sure about your proposal about how to test it. If responses on the two scales turned out to *not* be correlated, then that does seem to indicate that participants treat them differently.

However, if the two were highly correlated, matters might be less clear. That result seems compatible with a number of hypotheses, such as:

(1) People are plotting their response to the morality question based on how they'd answer the confidence question. (I float this idea in the paper of mine I link to in the post.)

(2) The reverse: people are treating the claims about confidence as about degrees of morality. (I've heard someone propose this in conversation, not about moral judgments but about knowledge attributions.)

(3) Participants treat the two as asking different questions but just happen to respond similarly. (But perhaps this hypothesis could be discarded as too implausible if the correlation holds up consistently.)

Anyway, as I say, it seems quite appropriate to tackle some of these issues experimentally, which is also in the spirit of John's idea that it's good to probe further once we find an initial result on whatever scale was used.


Thanks Josh.
I was struggling with my questionnaire. You saved my life with this post.
Sincerely yours

Brad Cokelet

Hi Josh,

Just a follow on the degrees issue. Degrees of wrongness can be motivated by thinking of various actions and comparing their apparent degree of wrongness. Some violations of moral obligation, for example, just do seem worse than others. Now this suggests that some obligations are more *important* than others and that violations of obligations can vary when it comes to their *moral seriousness*. But I do not see how this line of thought will support the idea that obligations or permissions themselves come in degrees. The idea that one obligation violations is worse than another gives no support to the idea that one can be more or less obligated or more or less permitted. So I think you would need to provide a new argument to motivate the later idea (and I am not seeing one right now).

Brad Cokelet

I see what you say about my previous categories, but I wonder whether people are running experiments in deontic and non-deontic categories and comparing the results? (real question!)

For example one could have this, in addition to the version that asks about permissiblity:

(a) I lean towards thinking this action is:

1: Morally Evil
5: Morally indifferent
10: Morally Admirable

(b) How Confident are you in your judgement?

Taylor Murphy

Josh, I agree that these issues about the importance of small variations in wording of the question are important. How else to tell whether two questions are in fact asking different things in empirical studies?

As for your 3 hypotheses, I have two other possible explanations, which are not mutually exclusive.

(4) Simple correlation: The more wrong some type of action is, the more overriding concerns are required to make it permissible in a concrete situation, and vice versa, and people have learned this. Murder is super wrong and so rarely permissible, and we can be super confident that any particular act of murder is wrong because exceptions are unlikely. Here I worry I am just ignorant of the relevant literature, but it does not seem impossible. Is there anything that is extremely wrong like murdering babies but in fact is almost always the right thing to do?

(5) Same mechanism: The same cognitive systems produce both judgments. There is a distinction between mechanisms underlying evaluations of certainty and subjective value and in the neuroeconomics literature on discounting, which might be of use. (The discounting literature focuses on temporal discounting, but some also consider probabilistic discounting.) If moral certainty and moral wrongness use the same system, it would be interesting that people tend to process both questions as valuation questions. How exactly to test such questions is unclear, but it might be possible to link the test to those already done on probabilistic discounting and valuation in the non-moral domain. Moral certainty : rightness :: reward certainty : subjective value.

Josh May

Brad, about degrees:
I think you're exactly right. It sounds like you read me as defending the idea that rightness/wrongness is gradable, but on the contrary I meant to be expressing serious doubts about it. But I was trying to take the opposing idea seriously because people like Hurka (and others in conversation) seem to. Maybe you're right, though, that there isn't even a way to motivate it! I suppose one could point to the fact that participants in the relevant studies worked with a scale just fine. But then of course the other worries kick in.

The comments to this entry are closed.