= 27d567bc2d48d9f40e9ccc20ea370b15)
lukeprog comments on Thoughts on the Singularity Institute (SI) - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (1270)
Though it's not as detailed and technical as many would like, I'll point readers to this bit of related reading, one of my favorites:
Yudkowsky (2011). Complex value systems are required to realize valuable futures.
It says:
No doubt a Martian Yudkowsy would make much the same argument - but they can't both be right. I think that neither of them are right - and that the conclusion is groundless.
Complexity theory shows what amazing things can arise from remarkably simple rules. Values are evidently like that - since even "finding prime numbers" fills the galaxy with an amazing, nanotech-capable spacefaring civilization - and if you claim that a nanotech-capable spacefaring civilization is not "interesting" you severely need recalibrating.
To end with, a quote from E.Y.:
I think Martian Yudkowsky is a dangerous intuition pump. We're invited to imagine a creature just like Eliezer except green and with antennae; we naturally imagine him having values as similar to us as, say, a Star Trek alien. From there we observe the similarity of values we just pushed in, and conclude that values like "interesting" are likely to be shared across very alien creatures. Real Martian Yudkowsky is much more alien than that, and is much more likely to say
Imagine, an intelligence that didn't have the universal emotion of badweather!
I suggest you guys taboo interesting, because I strongly suspect you're using it with slightly different meanings. (And BTW, as a Martian Yudkowsky I imagine something with values at least as alien as Babyeaters' or Superhappys'.)
It's another discussion, really, but it sounds as though you are denying the idea of "interestingness" as a universal instrumental value - whereas I would emphasize that "interestingness" is really just our name for whether something sustains our interest or not - and 'interest' is a pretty basic functional property of any agent with mobile sensors. There'll be other similarities in the area too - such as novelty-seeking. So shared common ground is only to be expected.
Anyway, I am not too wedded to Martian Yudkowsky. The problematical idea is that you could have a nanotech-capable spacefaring civilization that is not "interesting". If such a thing isn't "interesting" then - WTF?
Yes, I am; I think that the human value of interestingness is much, much more specific than the search space optimization you're pointing at.
[This reply was to an earlier version of timtyler's comment]
So: do you really think that humans wouldn't find a martian civilization interesting? Surely there would be many humans who would be incredibly interested.
I find Jupiter interesting. I think a paperclip maximizer (choosing a different intuition pump for the same point) could be more interesting than Jupiter, but it would generate an astronomically tiny fraction of the total potential for interestingness in this universe.
Life isn't much of an "interestingness" maximiser. Expecting to produce more than a tiny fraction of the total potential for interestingness in this universe seems as though it would be rather unreasonable.
I agree that a paperclip maximiser would be more boring than an ordinary entropy-maximising civilization - though I don't know by how much - probably not by a huge amount - the basic problems it faces are much the same - the paperclip maximiser just has fewer atoms to work with.
The goal "finding prime numbers" fills the galaxy with an amazing, nonotech-capable spacefaring network of computronium which finds prime numbers, not a civilization, and not interesting.
Maybe we should taboo the term interesting? My immediate reaction was that that sounded really interesting. This suggests that the term may not be a good one.
Fair enough. By "not interesting", I meant it is not the sort of future that I want to achieve. Which is a somewhat ideosyncratic usage, but I think inline with the context.
What if we added a module that sat around and was really interested in everything going on?
Not just computronium - also sensors and actuators - a lot like any other cybernetic system. There would be mining, spacecraft caft, refuse collection, recycling, nanotechnology, nuclear power and advanced machine intelligence with planning, risk assessment, and so forth. You might not be interested - but lots of folk would be amazed and fascinated.
Why?
If using another creature's values is effective at producing something "interesting", then 'detailed inheritance from human values' is clearly not needed to produce this effect.
So you're saying Earth Yudkowsky (EY) argues:
and Mars Yudkowsky (MY) argues:
and that one of these things has to be incorrect? But if martian and human values are similar, then they can both be right, and if martian and human values are not similar, then they refer to different things by the word "interesting".
In any case, I read EY's statement as one of probability-of-working-in-the-actual-world-as-it-is, not a deep philosophical point - "this is the way that would be most likely to be successful given what we know". In which case, we don't have access to martian values and therefore invoking detailed inheritance from them would be unlikely to work. MY would presumably be in an analogous situation.
I was assuming that 'detailed inheritance from human values' doesn't refer to the same thing as "detailed inheritance from martian values".
Maybe - but humans not finding martians interesting seems contrived to me. Humans have a long history of being interested in martians - with feeble evidence of their existence.
Right - so, substitute in "dolphins", "whales", or another advanced intelligence that actually exists.
Do you actually disagree with my original conclusion? Or is this just nit-picking?
I actually disagree that tiling the universe with prime number calculators would result in an interesting universe from my perspective (dead). I think it's nonobvious that dolphin-CEV-AI-paradise would be human-interesting. I think it's nonobvious that martian-CEV-AI-paradise would be human-interesting, given that these hypothetical martians diverge from humans to a significant extent.
I think it's violating the implied premises of the thought experiment to presume that the "interestingness evaluator" is dead. There's no terribly-compelling reason to assume that - it doesn't follow from the existence of a prime number maximizer that all humans are dead.
I may have been a little flip there. My understanding of the thought experiment is - something extrapolates some values and maximizes them, probably using up most of the universe, probably becoming the most significant factor in the species' future and that of all sentients, and the question is whether the result is "interesting" to us here and now, without specifying the precise way to evaluate that term. From that perspective, I'd say a vast uniform prime-number calculator, whether or not it wipes out all (other?) life, is not "interesting", in that it's somewhat conceptually interesting as a story but a rather dull thing to spend most of a universe on.
Today's ecosystems maximise entropy. Maximising primeness is different, but surely not greatly more interesting - since entropy is widely regarded as being tedious and boring.