= 27d567bc2d48d9f40e9ccc20ea370b15)
cousin_it comments on Complexity based moral values. - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (100)
Physics can contain objects whose complexity is much higher than that of physics. Do you have a strong argument why randomness didn't make a big contribution to human morality?
Well, suppose I were to make just the rough evolution sim, given really powerful computer. Even if it evolves society with principles we can deem moral once in a trillion societies - which is probably way low given that much of our principles are game theoretic - that just adds 40 bits to description for indexing those sims. edit: and the idea of the evolution sim doesn't really have such a huge complexity; any particular evolution sim does, but we don't care which evolution simulator we are working with; we don't need the bits for picking one specific one, just the bits for picking a working one.
Game-theoretic principles might be simple enough, but the utility function of a FAI building a good future for humanity probably needs to encode other information too, like cues for tasty food or sexual attractiveness. I don't know any good argument why this sort of information should have low complexity.
You may be over-fitting there. The FAI could let people decide what they want when it comes to food and attractiveness. Actually it better would, or i'd be having some serious regrets about this FAI.
That's reasonable, but to let people decide, the FAI needs to recognize people, which also seems to require complexity...
If your biggest problem is on the order of recognizing people, the problem of FAI becomes much, much easier.
Well, and the uFAI needs to know what "paperclips or something" means (or a real world goal at all). Obstacle faced by all contestants in the race. We humans learn what is other people and what isn't. (Or have evolved it, doesn't matter)
If you get paperclips slightly wrong, you get something equally bad (staples is the usual example, but the point is that any slight difference is about equally bad), but if you get FAI slightly wrong, you don't get something equally good. This breaks the symmetry.
I think if you get paperclips slightly wrong, you get a crash of some kind. If I get a ray-tracer slightly wrong, it doesn't trace electrons instead of photons.
edit: To clarify. It's about definition of person vs definition of paperclip. You need a very broad definition of person for FAI, so that it won't misidentify a person as non-person (misidentifying dolphins as persons won't be a big problem), and you need a very narrow definition of paperclip for uFAI, so that a person holding two papers together is not a paperclip. It's not always intuitive how broad definitions compare to narrow in difficulty, but it is worth noting that it is ridiculously hard to define paperclip making so that a Soviet factory anxious to maximize the paperclips would make anything at all, while it wasn't particularly difficult to define what a person is (or to define what 'money' are so that capitalist paperclip factory would make paperclips to maximize profit).
I agree that paperclips could also turn out to be pretty complex.
I don't think "paperclip maximizer" is taken as a complete declarative specification of what a paperclip maximizer is, let alone what it understands itself to be.
I imagine the setup is something like this. An AI has been created by some unspecified (and irrelevant) process and is now doing things to its (and our) immediate environment. We look at the things it has done and anthropomorphize it, saying "it's trying to maximize the quantity of paperclips in the universe". Obviously, almost every word in that description is problematic.
But the point is that the AI doesn't need to know what "paperclips or something" means. We're the ones who notice that the world is much more filled with paperclips after the AI got switched on.
This scenario is invariant under replacing "paperclips" with some arbitrary "X", I guess under the restriction that X is roughly at the scale (temporal, spatial, conceptual) of human experience. Picking paperclips, I assume, is just a rhetorical choice.
Well, I agree. That goes also for the what ever process determines something to be person. The difference is that the FAI doesn't have to create persons; it's definition doesn't need to process correctly things from the enormous space of possible things that can be or not be persons. It can have very broad definition that will include dolphins, and it will still be OK.
The intelligence, to some extent, is self defeating when finding a way to make something real; the easiest Y that is inside set X should be picked, by design, as instrumental to making more of some kind of X.
I.e. you define X to be something to hold papers together, the AI thinks and thinks and sees that a single atom, under some circumstances common in the universe (very far away in space), can hold the papers together; it finds the Kasimir effect which makes a vacuum able to hold two conductive papers together; and so on. The X has to be resistant against such brute forcing for the optimum solution.
Whenever the AI can come up with some real world manufacturing goal that it can't defeat in such a fashion, well, that's open to debate. Incomputable things seem hard to defeat.
edit: Actually. Would you consider a case of a fairly stupid nano-manufacturing AI destroying us, and itself, with gray goo, an unfriendly AI? That seems to be a particularly simple failure mode for self improving system, FAI or UFAI, under bounded computational power.And a failure mode for likely non-general AIs, as we are likely to employ such AIs to work on biotechnology and nanotechnology.
It doesn't sound like you are agreeing with me. I didn't make any assumptions about what the AI wants or whether its instrumental goals can be isolated. All I supposed was that the AI was doing something. I particularly didn't assume that the AI is at all concerned with what we think it is maximizing, namely, X.
As for the grey goo scenario, I think that an AI that caused the destruction of humanity not being called unfriendly would indicate a incorrect definition of at least one of "AI", "humanity", or "unfriendly" ("caused" too, I guess).
Can you be more specific? I have an AI that's iterating parameters to some strange attractor - defined within it - until it finds unusual behaviour. I can make the AI that would hillclimb+search for the improvements to the former AI. edit: Now, the worst thing that can happen, it makes mind hack image that kills everyone who looks at it. That wasn't the intent, but the 'unusual behaviour' might get too unusual for human brain to handle. Is that a serious risk? No it's a laughable one.
...or perhaps "destruction".
Randomness is Chaitin's omega is God implies stochasticity (mixed Strategies) implies winning in the limit due to hypercomputational advantages universally if not necessarily contingently. Hence randomness isn't at odds as such with morality. Maybe Schmidhuber's ideas about super-omegas are relevant. Doubt it.