Quinn comments on Moloch: optimisation, "and" vs "or", information, and sacrificial ems - Less Wrong

20 Post author: Stuart_Armstrong 06 August 2014 03:57PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (58)
Tags: research optimisation optimization moloch project

Comments (58)

You are viewing a single comment's thread.

Comment author: Quinn 11 August 2014 02:50:06AM 15 points [-]

Because the length of Scott's Moloch post greatly exceeds my working memory (to the extent that I had trouble remembering what the point was by the end) I made these notes. I hope this is the right place to share them.

Notes on Moloch (ancient god of child sacrifice)

http://slatestarcodex.com/2014/07/30/meditations-on-moloch/

  1. Intro - no real content.

  2. Moloch as coordination failure: everyone makes a sacrifice to optimize for a zero-sum competition, ends up with the same relative status, but worse absolute status.

    • 10 examples: Prisoner's Dilemma, dollar auctions, fish-farming story (tragedy of the commons), Malthusian trap, ruthless/exploitative Capitalist markets, the two-income trap, agriculture, arms races, cancer, political race to the bottom (lowering taxes to attract business)
    • 4 partial examples: inefficient education, inefficient science, government corruption (corporate welfare), Congress (representatives voting against good of nation for good of constituency)

  3. Existing systems are created by incentive structures, not agents, e.g. Las Vegas caused by a known bias in human reward circuitry, not optimization for human values.

  4. But sometimes we move uphill anyway. Possible explanations:

    • Excess resources / we are in the dream time and can afford non-competitive behavior.
    • Physical limitations to what can be sacrificed
    • Economic competition actually producing positive utility for consumers (but this is fragile)
    • Coordination, e.g. via governments, guilds, friendships, etc.

  5. Technology/ingenuity creates new opportunities to fall into such traps. Technology overcomes physical limitations, consumes excess resources. Automation further decouples economic activity from human values. Technology can improve coordination, but can also exacerbate existing conflicts by giving all sides more power.

AGI opens up whole new worlds of traps: Yudkowsky's paperclipper, Hanson's subsistence-level ems, Bostrom's Disneyland with no children.

6 & 7. Gnon - basically the god of the conservative scarcity mindset. Nick Land advocates compliance; Nyan wants to capture Gnon and build a walled garden. Scott warns that Moloch is far more terrifying than Gnon and will kill both of them anyway.

8 & 9. So we have to kill this Moloch guy, by lifting a better God to Heaven (Elua).

Comment author: lukeprog 17 August 2014 03:45:52PM 6 points [-]

everyone makes a sacrifice to optimize for a zero-sum competition, ends up with the same relative status, but worse absolute status.

I'm a bit surprised I haven't seen this particular incentives problem named in the academic literature. It is related in different ways to economic concepts like tragedy of the commons, social trap, tyranny of small decisions, and information asymmetry, but isn't identical with or fully captured by any of them.

Comment author: Wei_Dai 18 August 2014 04:55:38AM 6 points [-]

See also, positional good, which pre-dates Robert Frank's "positional arms race".

Comment author: lukeprog 24 August 2014 03:29:10AM 4 points [-]

Thanks. That led me to Robert Frank on positional externalities.

Comment author: satt 18 August 2014 02:04:14AM *  5 points [-]

everyone makes a sacrifice to optimize for a zero-sum competition, ends up with the same relative status, but worse absolute status.

I'm a bit surprised I haven't seen this particular incentives problem named in the academic literature.

Robert H. Frank has called it a "positional arms race". In a relatively recent article on higher education he gives this summary:

Participants in virtually all winner-take-all markets face strong incentives to invest in performance enhancement, thereby to increase their chances of coming out ahead. As in the classic military arms race, however, many such investments prove mutually offsetting in the end. When each nation spends more on bombs, the balance of power is no different than if none had spent more. Yet that fact alone provides no escape for individual participants. Countries may find it burdensome to spend a lot on bombs, but the alternative—to be less well-armed than their rivals—is even worse.

In light of the growing importance of rank in the education marketplace, universities face increasing pressure to bid for the various resources that facilitate the quest for high rank. These pressures have spawned a positional arms race that already has proved extremely costly, and promises to become more so.

Comment author: lukeprog 18 August 2014 03:16:50AM 5 points [-]

Cool, thanks for the pointer!

Comment author: kpreid 04 September 2014 03:26:38AM 4 points [-]

everyone makes a sacrifice to optimize for a zero-sum competition, ends up with the same relative status, but worse absolute status

Isn't a competition in which that outcome is possible not zero-sum, by definition?

Comment author: Stuart_Armstrong 11 August 2014 10:24:00AM 4 points [-]

Yep, a fair summary, with none of the wild poetry :-)

Comment author: Clarity 05 March 2016 03:50:19AM 0 points [-]

I came here to complain that I don't understand meditations on moloch at all. Can someone explain the points more explicitly? And you have. Thank you.

If:

Excess resources / we are in the dream time and can afford non-competitive behavior. Physical limitations to what can be sacrificed Economic competition actually producing positive utility for consumers (but this is fragile) Coordination, e.g. via governments, guilds, friendships, etc.

Then those factors are the critical systemic elements of progress.

However, is there any reason to believe those are those systemic elements, rather than others?

Powered by Reddit Powered by Reddit