AIs, Honesty, Murder and Trolleys

[Johnny1A.2]

Originally Posted by Flyndaran

War messes people up, even though for some reason I don't understand, killing innocents is more acceptable then. Probably a big part of it is diffusion of guilt that wouldn't apply to OP's scenario.

All moral codes contain a-rational elements. These can't be expressed by purely mathematical equivalencies, but they can't be escaped, either.

That's directly on-topic because the nature of an AI's moral code, whether self-originated or programmed, will depend in part on those a-rational elements. If the AI isn't able to make such distinctions, there's a good chance that humans would see that as proof that it isn't really a person, or at least not a person that can be trusted to act as a member of a human society.

(In fact, that's an old SFnal trope, the computer that follows the moral code programmed into it without really understanding the 'soft' or a-rational elements of it, and produces results the creators would find horrifying by doing so.)

I can cite a fictional example of the effect I'm talking about, for illustration, from the old movie Stargate. At one point in the movie, there's a scene were a Marine is about to gun down the chief villain (who is very villainous) but stops because a crowd of children loyal to the villain jump between them, and he can't bring himself to gun them down in cold blood.

(He had personal reasons for such hesitation, as well as moral ones.)

Later, though, he and the other protagonist teleport a nuke aboard the villain's spacecraft and blow it fine dust, children and all. It was the only alternative to letting the nuke go off where they were, killing not only they themselves but many of their Terran and non-Terran allies, and destroying their FTL link back to Earth.

It really was an either/or choice, but the Marine in question still killed the children, the sames ones he couldn't gun down in cold blood, and didn't appear to consider it unaccceptable. Various explanations for the mathematically equivalent actions could be posited, but they all come down at least in part to a-rational motivations and considerations. Most of those citable differences in the two situations can be played in reverse to show that gunning down the children was actually defensible, too, in purely 'mathematical' terms. Yet I suspect that most viewers would deny the equivalence on a-rational grounds.

Such considerations always play a role in moral codes.