Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

manuelg comments on Terminal Values and Instrumental Values - Less Wrong

54 Post author: Eliezer_Yudkowsky 15 November 2007 07:56AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (39)

Sort By: Old

You are viewing a single comment's thread.

Comment author: manuelg 15 November 2007 08:56:50PM 0 points [-]

The very first "compilation" I would suggest to your choice system would be to calculate the "Expected Utility of Success" for each Action.

1) It is rational to be prejudiced against Actions with a large difference between their "Expected Utility of Success" and their "Expected Utility", even if that action might have the highest "Expected Utility". People with a low tolerance for risk (constitutionally) would find the possible downside of such actions unacceptable.

2) Knowing the "Expected Utility of Success" gives information for future planning if success is realized. If success might be "winning a Hummer SUV in a raffle in December", it would probably be irrational to construct a "too small" car port in November, even with success being non-certain.

Eliezer, I have a question.

In a simple model, how best to avoid the failure mode of taking a course of action with an unacceptable chance of leading to catastrophic failure? I am inclined to compute separately, for each action, its probability of leading to a catastrophic failure, and immediately exclude from further consideration those actions that cross a certain threshold.

Is this how you would proceed?