Vaniver comments on Rationality Quotes February 2013 - Less Wrong

2 Post author: arundelo 05 February 2013 10:20PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (563)

You are viewing a single comment's thread. Show more comments above.

Comment author: Eliezer_Yudkowsky 06 February 2013 11:33:28PM 22 points [-]

I've just come across a fascinatingly compact observation by I. J. Good:

Public and private utilities do not always coincide. This leads to ethical problems. Example - an invention is submitted to a scientific adviser of a firm...

The probability that the invention will work is p. The value to the firm if the invention is adopted and works is V, and the loss if the invention is adopted and fails is L. The value to the adviser personally if he advises the adoption of the invention and it works is v, and the loss if it fails to work is l. The losses to the firm and the adviser if he recommends the rejection of the invention are both negligible...

Then the firm's expected gain if the invention is adopted is pV - (1-p)L and the adviser's expected gain in the same circumstances is pv - (1-p)l. The firm has positive expected gain if p/(1-p) > L/V, and the adviser has positive expected gain if p/(1-p) > l/v.

If l/v > p/(1-p) > L/V, the adviser will be faced with an ethical problem, i.e. he will be tempted to act against the interests of the firm.

This is a beautifully simple recipe for a conflict of interest:

Considering absolute losses assuming failure and absolute gains conditioned on success, an adviser is incentivized to give the wrong advice, precisely when:

  • The ratio of agent loss to agent gain,
  • exceeds the odds of success versus failure
  • which in turn exceeds the ratio of principal loss to principal gain.

You can see this reflected in a lot of cases because the gains to an advisor often don't scale anywhere near as fast as the gains to society or a firm. It's the Fearful Committee Formula.

Comment author: Vaniver 06 February 2013 11:39:09PM *  8 points [-]

In general, this is referred to as the principal-agent problem.

Note that the adviser's ethical problem also exists if L/V > p/(1-p) > l/v.

The adviser to the value

Is the order also inverted in the original?

Comment author: Eliezer_Yudkowsky 06 February 2013 11:48:02PM 1 point [-]

Fixed.

I. J. Good's original, which I've somewhat abridged, explicitly specifies that there are no competitors who cause visible losses/gains after the invention is rejected.

Comment author: Vaniver 06 February 2013 11:53:20PM 2 points [-]

I. J. Good's original, which I've somewhat abridged, explicitly specifies that there are no competitors who cause visible losses/gains after the invention is rejected.

To clarify, this is a summary of what you've excluded in your quote, not a response to the other case where the ethical problem exists, correct?

Comment author: Eliezer_Yudkowsky 07 February 2013 12:07:30AM 2 points [-]

It's a summary of what I excluded - I had actually misinterpreted, hence my quote indeed was not a valid reply! The other case is indeed real, sorry.