glomerulus comments on Tell Culture - Less Wrong

109 Post author: BrienneYudkowsky 18 January 2014 08:13PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (217)

You are viewing a single comment's thread. Show more comments above.

Comment author: BrienneYudkowsky 18 January 2014 08:11:48PM 2 points [-]

A community of HPMOR!Quirrell variations would have your very post in main, with plenty of upvotes, all the while secretly whetting their blades. Perfectly rational.

I really don't think so. A community of Briennes, which is not a community of HPMOR!Quirrells but shares some relevant features, would recognize the overwhelming benefit of coordination. Any given individual would be much stronger if she had the knowledge of all the other individuals, or if she could count on them as external memory. And because she would be stronger that way, she knows that they would be stronger if she also remains trustworthy. Her being trustworthy allows her to derive greater benefit from the rest of the community. Other people are useful, you see. With Tell culture in place, you can do things like feed your model of the world into someone else's truth-checker and get back a more info-rich version. You only defect if the expected utility of doing so outweighs the expected utility of the entire community to your future plans.

I'd love to hear what culture Eliezer thinks an entire community of Quirrells would create.

Comment author: glomerulus 19 January 2014 03:38:32AM 2 points [-]

"You only defect if the expected utility of doing so outweighs the expected utility of the entire community to your future plans." These aren't the two options available, though: you'd take into account the risk of other people defecting and thus reducing the expected utility of the entire community by an appreciable amount. Your argument only works if you can trust everyone else not to defect, too - in a homogenous community of Briennes, for instance. In a heterogenous community, whatever spooky coordination your clones would use won't work, and cooperation is a much less desirable option.