Will, you still seem to think Eliezer is saying "do what an optimal agent would, even when doing so is not optimal." No.

[Richard] Anyone can test this. Give someone directions in the form of a plan of which way to go at each intersection. Don't tell them the destination you're aiming them at. Have them execute the plan and tell you where they got to.

But then they're not optimizing for a goal they can be said to have. If a street is closed, they won't reach the destination, as they would if they knew what the destination was.