Comment author: benwr 26 March 2016 08:33:27AM 7 points [-]

Great survey!

However, when you save your progress and are asked to save a password, there's no indication that it will be sent to you in an email or saved at all in recoverable form. I used my least-secure password generation algorithm anyway, but: Do you think you could add a note to the effect that users should not use passwords that they use elsewhere?

Comment author: Alexandros 13 January 2015 06:22:01PM 4 points [-]

The truly insidious effects are when the content of the stories changes the reward but not by going through the standard quality-evaluation function.

For instance, maybe the AI figures out that the order of the stories affects the rewards. Or perhaps it finds how stories that create a climate of joy/fear on campus lead to overall higher/lower evaluations for that period. Then the AI may be motivated to "take a hit" to push through some fear mongering so as to raise its evaluations for the following period. Perhaps it finds that causing strife in the student union, or perhaps causing racial conflict, or causing trouble with the university faculty affects its rewards one way or another. Perhaps if it's unhappy with a certain editor, it can slip through bad enough errors to get the editor fired, hopefully replaced with a more rewarding editor.

etc etc.

Comment author: benwr 14 January 2015 02:34:54AM *  2 points [-]

The problem with these particular extensions is that they don't sound plausible for this type of AI. In my opinion it would be easier when talking with designers to switch from this example to a slightly more sci-fi example.

The leap is between the obvious "it's 'manipulating' its editors by recognizing simple patterns in their behavior" to "it's manipulating its editors by correctly interpreting the causes underlying their behavior."

Much easier to extend in the other direction first: "Now imagine that it's not an article-writer, but a science officer aboard the commercial spacecraft Nostromo..."

Comment author: Lalartu 12 January 2015 09:27:59AM 1 point [-]

A century ago there were scientists who said the same. Just because somebody is working on a problem doesn't mean it will be solved.

Comment author: benwr 12 January 2015 10:34:38AM 1 point [-]

On the other hand, the number of people working on a problem, and the speed with which they are individually able to work, can't be ignored. "Given enough eyes, all bugs are shallow" - Linus Torvalds, talking about something pretty similar (if much, much simpler).