I doubt there's all that much of a correlation between these things to be honest.

I'd find this more informative if you explicitly addressed my examples?

I mean something morally meaningful. I don't think a chess computer suffers when it loses a game, no matter how sophisticated. I expect that self-driving cars are programmed to try to avoid accidents even when other drivers drive badly, but I don't think they suffer if you crash into them.

I am inclined to believe that what we call "consciousness" and even "sentience" may turn out to be ideas fully as human-specific as Eliezer's favourite example, "humour".

There's at least a possibility that "suffering" is almost as specific.

If there's actual science here, that's even better, but I don't get the impression there is - each individual is their own experiment to some extent, but effects like the file drawer effect are likely to be pretty serious!

private_messaging says he's Dmytry, who has positive karma. It's possible that the more anonymous-sounding name encourages worse behaviour though.

Now that the huffy remark has been removed, I can't see what post it used to refer to!

Complete quote is

To speak of building an AGI which shares "our values" is likely to provoke negative reactions from any AGI researcher whose current values include terms for respecting the desires of future sentient beings and allowing them to self-actualize their own potential without undue constraint. This itself, of course, is a component of the AGI researcher's preferences which would not necessarily be shared by all powerful optimization processes, just as natural selection doesn't care about old elephants starving to death or gazelles dying in pointless agony. Building an AGI which shares, quote, "our values", unquote, sounds decidedly non-cosmopolitan, something like trying to rule that future intergalactic civilizations must be composed of squishy meat creatures with ten fingers or they couldn't possibly be worth anything - and hence, of course, contrary to our own cosmopolitan values, i.e., cosmopolitan preferences. The counterintuitive idea is that even from a cosmopolitan perspective, you cannot take a hands-off approach to the value systems of AGIs; most random utility functions result in sterile, boring futures because the resulting agent does not share our own intuitions about the importance of things like novelty and diversity, but simply goes off and e.g. tiles its future lightcone with paperclips, or other configurations of matter which seem to us merely "pointless".

Schmidthuber's definition of beauty is wrong. He says, roughly, that you're most pleased when after great effort you find a way to compress what was seemingly incompressible. If that were so, I could please you again and again by making up new AES keys with the first k bits random and the rest zero, and using them to generate and give you a few terabytes of random data. You'd have to brute force the key, at which point you'll have compressed down from terabytes to kilobytes. What beauty! Let's play the exact game again, with the exact same cipher but a different key, forever.