There’s this classic thought experiment that people use to argue against utilitarianism. You’re a doctor with five patients about to die for want of different organs, when you get one patient who’s in no danger of dying but is a match for all five of them. You can harvest his organs (which will kill him) and no one will ever know. Killing him seems correct under utilitarianism, but is it really the right thing to do?
I’ve heard two utilitarian responses to the thought experiment. One is that killing the guy is the right thing to do, and our moral intuitions, not designed for weird edge cases such as this, are leading us astray. The other is that killing the guy is wrong, because you’re contributing to a precedent of killing random people whenever they come to the hospital and that outweighs the four lives you’re saving.
But with this and this floating around my head recently I’ve settled on a third solution. I think that killing him does increase utility. But if everyone made the utility-increasing choice, then the world would be a worse place. It’s the altruistic version of the prisoner’s dilemma.
So how do we solve this? This is where moral rules like “keep your promises” or “don’t tell lies” or “don’t kill people even if you think it’s a really good idea” come in, to enforce coordination. Folk morality does a great job with this. The issue is that folk morality is a paperclip maximizer. It’s optimized for genetic and memetic fitness rather than any human values. And it still seems like utilitarians do better than most deontologists on the important questions. But this failure mode is kind of a problem.