Exactly. But you can come up with an much harsher example than aimlessly driving a car around:

In general it seems like destroying all other agents with potentially different optimization criteria would be have instrumental value, however, killing other people isn't, in general, right, even if, say, they're your political adversaries.

And again, I bet Roko didn't even consider "destroy all other agents" as a candidate UIV because of anthropomorphic optimism.

Incidentally Eliezer, is this really worth your time?

I thought the main purpose of your taking time off AI research to write overcoming bias was to write something to get potential AI programmers to start training themselves. Do you predict that any of the people we will eventually hire will have clung to a mistake like this one despite reading through all of your previous series of posts on morality?

I'm just worried that arguing of this sort can become a Lost Purpose.