Eliezer_Yudkowsky comments on Rationality Quotes February 2013 - Less Wrong

2 Post author: arundelo 05 February 2013 10:20PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (563)

You are viewing a single comment's thread.

Comment author: Eliezer_Yudkowsky 06 February 2013 11:33:28PM 22 points [-]

I've just come across a fascinatingly compact observation by I. J. Good:

Public and private utilities do not always coincide. This leads to ethical problems. Example - an invention is submitted to a scientific adviser of a firm...

The probability that the invention will work is p. The value to the firm if the invention is adopted and works is V, and the loss if the invention is adopted and fails is L. The value to the adviser personally if he advises the adoption of the invention and it works is v, and the loss if it fails to work is l. The losses to the firm and the adviser if he recommends the rejection of the invention are both negligible...

Then the firm's expected gain if the invention is adopted is pV - (1-p)L and the adviser's expected gain in the same circumstances is pv - (1-p)l. The firm has positive expected gain if p/(1-p) > L/V, and the adviser has positive expected gain if p/(1-p) > l/v.

If l/v > p/(1-p) > L/V, the adviser will be faced with an ethical problem, i.e. he will be tempted to act against the interests of the firm.

This is a beautifully simple recipe for a conflict of interest:

Considering absolute losses assuming failure and absolute gains conditioned on success, an adviser is incentivized to give the wrong advice, precisely when:

  • The ratio of agent loss to agent gain,
  • exceeds the odds of success versus failure
  • which in turn exceeds the ratio of principal loss to principal gain.

You can see this reflected in a lot of cases because the gains to an advisor often don't scale anywhere near as fast as the gains to society or a firm. It's the Fearful Committee Formula.

Comment author: shminux 07 February 2013 12:04:51AM *  12 points [-]

the Fearful Committee Formula.

Which is not nearly as common as the reverse, the Reckless Adviser Formula, when the personal loss to the adviser is so low and the potential personal gain is so high, they recommend adoption even when the expected gain for the company is negative.

Comment author: Vaniver 06 February 2013 11:39:09PM *  8 points [-]

In general, this is referred to as the principal-agent problem.

Note that the adviser's ethical problem also exists if L/V > p/(1-p) > l/v.

The adviser to the value

Is the order also inverted in the original?

Comment author: Eliezer_Yudkowsky 06 February 2013 11:48:02PM 1 point [-]

Fixed.

I. J. Good's original, which I've somewhat abridged, explicitly specifies that there are no competitors who cause visible losses/gains after the invention is rejected.

Comment author: Vaniver 06 February 2013 11:53:20PM 2 points [-]

I. J. Good's original, which I've somewhat abridged, explicitly specifies that there are no competitors who cause visible losses/gains after the invention is rejected.

To clarify, this is a summary of what you've excluded in your quote, not a response to the other case where the ethical problem exists, correct?

Comment author: Eliezer_Yudkowsky 07 February 2013 12:07:30AM 2 points [-]

It's a summary of what I excluded - I had actually misinterpreted, hence my quote indeed was not a valid reply! The other case is indeed real, sorry.

Comment author: Qiaochu_Yuan 06 February 2013 11:36:16PM 5 points [-]

You can see this reflected in a lot of cases because the gains to an advisor often don't scale anywhere near as fast as the gains to society or a firm.

Name three?

Comment author: Vaniver 06 February 2013 11:44:29PM 7 points [-]

The success of Market-Based Management / Koch Industries appears to be due at least in part to their focus on NPV at the managerial level. You get stories like (from memory, and thus subject to fuzz) the manager of a refining plant selling the land the plant was on to a casino which was moving to the area, which he was rewarded for doing because the land the plant was on was more valuable to the casino than the company, even after factoring in the time lost because the plant was shut down and relocated. The corporate culture (and pay incentive structure) rewarded that sort of lateral use of resources, whereas a culture which compartmentalized people and departments would have balked at the lost time and disruption.