= 27d567bc2d48d9f40e9ccc20ea370b15)
Major AI risk is science fiction -- that is, it's the kind of thing science-fiction stories get written about, and it isn't something we have experience of yet outside fiction. I don't see how any thought experiment that seriously engages with the issue could not pattern-match to science fiction.
I think many people intuitively distrust the idea that an AI could be intelligent enough to transform matter into paperclips in creative ways, but 'not intelligent enough' to understand its goals in a human and cultural context (i.e. to satisfy the needs of the business owners of the paperclip factory). This is often due to the confusion that the paperclip maximizer would get its goal function from parsing the sentence "make paperclips", rather from a preprogrammed reward function, for example a CNN that is trained to map the number of paperclips in images to a scalar reward.
In response to
Torture vs. Dust Specks
I think the problem here is the way the utility function is chosen. Utilitarianism is essentially a formalization of reward signals in our heads. It is a heuristic way of quantifying what we expect a healthy human (one that can raise up and survive in a typical human environment and has an accurate model of reality) to want. All of this only converges roughly to a common utility because we have evolved to have the same needs which are necessarily pro-life and pro-social (since otherwise our species wouldn't be present today).
Utilitarianism crudely abstracts from the meanings in our heads that we recognize as common goals and assigns numbers to them. We have to be careful what we want to assign numbers to in order to get results that we want in all corner cases. I think, hooking up the utility meter to neurons that detect minor inconveniences is not a smart way of achieving what we collectively want because it might contradict our pro-life and pro-social needs. Only when the inconveniences accumulate individually so that they condense as states of fear/anxiety or noticeably shorten human life, it affects human goals and it makes sense to include them into utility considerations (which, again, are only a crude approximation of what we have evolved to want).
In response to
Open Thread, January 4-10, 2016
Why does E. Yudkowsky voice such strong priors e.g. wrt. the laws of physics (many worlds interpretation), when much weaker priors seem sufficient for most of his beliefs (e.g. weak computationalism/computational monism) and wouldn't make him so vulnerable? (With vulnerable I mean that his work often gets ripped apart as cultish pseudoscience.)
In response to
Rationality Reading Group: Part Q: Joy in the Merely Real
I would love to seem some hard data about correlation between the public interest in science and it's degree of 'cult status' vs. 'open science'.
"Only a meme" doesn't negate that it's about something real and that there can be resonable arguments why some memes are better than others.
I mean "only a meme" in the sense, that morality is not absolute, but an individual choice. Of course, there can be arguments why some memes are better than others, that happens during the act of individuals convincing each other of their preferences.
Basically your argument is: "I can't think of a way to justify morality besides saying that it's my own prefered state, therefore nobody can come up with an argument to justify morality."
Is it? I think, the act of convincing other people of your preferred state of the world is exactly what justifying morality is. But that action policy is only a meme, as you said, which is individually chosen based on many criteria (including aesthetics, peer-pressure, consistency).
What do you mean with should?
Moral philosophy is a huge topic and it's discourse is not dominated by looking at DNA.
Everyone can choose their preferred state then, at least to the extent it is not indoctrinated or biologically determined. It is rational to invest energy into maintaining or achieving this state (because the state presumably provides you with a steady source of reward), which might involve convincing others of your preferred state or prevent them from threating it (e.g. by putting them into jail). There is likely an absolute truth (to the extent physics is consistent from our point of view), but no absolute morale (because it's all memes in an undirected process). Terrorists do nothing wrong from their point of view, but from mine it threatens my preferred state, so I will try to prevent terrorism. We may seem lucky that many preferred states converge to the same goals which are even fairly sustainable, but that is just an evolutionary necessity and perhaps mostly a result of empathy and the will to survive (otherwise our species wouldn't have survived in paleolithic groups of hunters and gatherers).
Our culture is just as backed into us as our DNA. It's all memes.
What are the implications of that on how we decide what is are the right things to do?
Why do you think biology basis has something to do with the answer?
Because then it would argue from features that are built into us. If we can prove the existence of these features with high certainty, then it could perhaps serve as guidance for our decisions.
On the other hand, it is reasonable that evolution does not create such goals because it is an undirected process. Our actions are unrestricted in this regard, and we must only bear the consequences of the system that our species has come up with. What is good is thus decided by consensus. Still, the values we have converged to are shaped by the way we have evolved to behave (e.g. empathy and pain avoidance).
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Could well be. Does that have anything to do with pattern-matching AI risk to SF, though?
Just speaking of weaknesses of the paperclip maximizer though experiment. I've seen this misunderstanding at least 4 out of 10 times that the thought experiment was brought up.