Less Wrong: Open Thread, September 2010

matt

This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.

A paperclip maximizer also has a dead-end goal - maximizing paperclips - which is what I object to. An non-Friendly AI just has no particular interest in humans; there is no other necessary claim you can make about it.

You have merely redefined the goal from 'the benefit of humanity' to 'non dead-end goal', which may just be equally hairy.

2Snowyowl16y

Ah, I see. I misunderstood your definition of "paperclip maximiser"; I assumed paperclip maximiser and Unfriendly AI were equivalent. Sorry. Next question: if maximising paperclips or relentless self-optimisation is a dead-end goal, what is an example of a non-dead-end goal? Is there a clear border between the two, which will be obvious to the AI? To my mind, if paperclip maximisation is a dead end, then so is everything else. The Second Law of Thermodynamics will catch up with you eventually. Nothing you create will endure forever. The only thing you can do is try to maximise your utility for as long as possible, and if that means paperclips, then so be it.

2Oscar_Cunningham16y

What would be a non-friendly goal that isn't dead-end? (N.B. Not a rhetorical question.)