= 27d567bc2d48d9f40e9ccc20ea370b15)
jhuffman comments on Complexity of Value ≠ Complexity of Outcome - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (198)
Can we really expect a superintelligence to stick with the values we give it ? Our own values change over time; sometimes without even external stimulus just internal reflection. I don't see how we can bound a superintelligence without doing more computation than we expect it to do in its lifetime.
I tend to file this under "humans are stupid." Messy creatures like ourselves undergo value drift, but decision-theoretically speaking, systems designed to optimize for some particular criterion have a natural incentive to keep that criterion. Cf. "The Basic AI Drives."
It is probably best to model those as infections - or sometimes malfunctions.
Humans get infected with pathogens that make them do things like sneeze. Their values have not changed to value spreading snot on their neigbours, rather they are infected with germs - and the germs do value that.
It's much the same with mind-viruses. A catholic conversion is best modelled as a memetic infection - rather than a genuine change in underlying values. Such people can be cured.
The fact that a change is reversible does not make it not real.
The fact that the final value system can be modeled as a starting value system modified by "memetic infection" does not make the final value system invalid. They are two different but equivalent ways of modelling the state.
Right. The point is that - under the "infection" analogy - people's "ultimate" values change a lot less. How much they change depends on the strength of people's memetic immune system - and there are some people with strong memetic immune systems whose values don't change much at all.
I'm not sure I follow you.
Are you saying that some agents change their values less often than others (or equivalently, are less likely to acquire "infections")?
Yes.
I once proved that a program will print out only prime numbers endlessly. I really, really wish I kept the working out.
Is that program still running? ;-)
Hush you. You weren't supposed to notice that. :D
Quite a bit of ink has been spilled on this issue. Eliezer Yudkowsky and Steve Omohundro have argued that it is possible. Have you examined their arguments?