= 27d567bc2d48d9f40e9ccc20ea370b15)
Roko comments on Open Thread: February 2010, part 2 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (857)
The reason I think it can just be a one-time shock is that we can extend our preferences to cover all possible mathematical structures. (I talked about this in Towards a New Decision Theory.) Then, no matter what kind of universe we turn out to live in, whichever theory of quantum gravity turns out to be correct, the structure of the universe will correspond to some mathematical structure which we will have well-defined preferences over.
I addressed this issue a bit in that post as well. Are you not convinced that rational decision-making is possible in Tegmark's Level IV Multiverse?
The next few posts on my blog are going to be basically about approaching this problem (and given the occasion, I may as well commit to writing the first post today).
You should read [*] to get a better idea of why I see "preference over all mathematical structures" as a bad call. We can't say what "all mathematical structures" is, any given foundation only covers a portion of what we could invent. As the real world, mathematics that we might someday encounter can only be completely defined by the process of discovery (but if you capture this process, you may need nothing else).
--
[*] S. Awodey (2004). `An Answer to Hellman's Question: 'Does Category Theory Provide a Framework for Mathematical Structuralism?". Philosophia Mathematica 12(1):54-64.
Hope to finish it today... Though I won't talk about philosophy of mathematics in this sub-series, I'm just going to reduce the ontological confusion about preference and laws of physics to a (still somewhat philosophical, but taking place in a comfortably formal setting) question of static analysis of computer programs.
Great to hear. Looking forward to reading it.
Yes, talking about "preference over all mathematical structures" does gloss over some problems in the philosophy of mathematics, and I am sympathetic to anti-foundationalist views like Awodey's.
Also, in general I agree with Roko on the need for an AI that can do philosophy better than any human, so in this thread I was mostly picking a nit with a specific argument that he had.
(I was going to remind you about the missing post, but I see Roko already did. :)
I disagree on the first part, and agree on the second part.
Yes, and that's enough for rational decision making. I'm not really sure why you're not seeing that...
There is a deep analogy between how you can't change the laws of physics (contents of reality, apart from lawfully acting) and how you can't change your own program. It's not a delusion unless it can be reached by mistake. The theist can't be right to act as if a deity exists unless his program (brain) is such that it is the correct way to act, and he can't change his mind for it to become right, because it's impossible to change one's program, only act according to it.
Following the same analogy, you can translate it as "if only the God did in fact exist, ...". The difference doesn't seem particularly significant -- both "what ifs" are equally impossible. "Regretting rationality" is on a different level -- rationality in the relevant sense is a matter of choice. The program that defines your decision-making algorithm isn't.
I still fear that you are reading in my words something very different from what I intend, as I don't see the possibility of a religious person's mind actually acting as if God is real. A religious person may have a free-floating network of beliefs about God, but it doesn't survive under reflection. A true god-impressed mind would actually act as if God is real, no matter what, it won't be deconvertable, and indeed under reflection an atheist god-impressed mind will correctly discard atheism.
Not all beliefs are equal, a human atheist is correct not just according to atheist's standard, and a human theist is incorrect not just to atheist's standard. The standard is in the world, or, under this analogy, in the mind. (The mind is a better place for ontology, because preference is also here, and human mind can be completely formalized, unlike the unknown laws of physics. By the way, the first post is up).
But only in the sense that their calculation could be correct according to a particularly weird prior. The difference between normal theist and a "god-impressed mind" who both believe in God is that of rationality: the former makes mistakes in updating beliefs, the latter probably doesn't. The same with an atheist god-impressed mind and a human atheist. You can't expect to find that weird a prior in a human. And of course, you should say that the god-impressed are wrong about their beliefs, though they correctly follow the evidence according to their prior. If you value their success in the real world more than the autonomy of their preference, you may want to reach into their minds and make appropriate changes.
I should say again: the program that defines the decision-making algorithm can't be normally changed, which means that one can't be really "converted" to a different preference, though one can be converted to different beliefs and feelings. Observations don't change the algorithm, they are processed according to that algorithm. This means that if you care about reflective consistency (and everyone does, in the sense of preservation of preference), you'd try to counteract the unwanted effects of environment on yourself, including the self-promoting effects where you start liking the new situation. The extent to which you like the new situation, the "level of conviction", it's pretty much irrelevant, just as the presence of a losing psychological drive. It'd take great integrity (not "strength of conviction") in the change for significantly different values to really sink in, in the sense that the new preference-on-reflection will resemble the new beliefs and feelings similarly to how the native preference-on-reflection will resemble native (sane, secular, etc.) beliefs and feelings.
I agree that it's ugly to think of the weights as a pretense on how real certain parts of reality are. That's why I think it may be better to think of them as representing how much you care about various parts of reality. (For the benefit of other readers, I talked about this in What Are Probabilities, Anyway?.)
Actually, I haven't completely given up the idea that there is some objective notion of how real, or how important, various parts of reality are. It's hard to escape the intuition that some parts of math are just easier to reach or find than others, in a way that is not dependent how human minds work.
I believe even personal identity falls under this category. A lot of moral intuitions work with the-me-in-the-future object, as marked in the map. To follow these intuitions, it is very important for us to have a good idea of where the-me-in-the-future is, to have a good map of this thing. When you get to weird thought experiments with copying, this epistemic step breaks down, because if there are multiple future-copies, the-me-in-the-future is a pattern that is absent. As a result, moral intuitions, that indirectly work through this mark on the map, get confused and start giving the wrong answers as well. This can be readily observed for example from preferential inconsistency in time expected in such thought experiments (you precommit to teleporting-with-delay, but then your copy that is to be destroyed starts complaining).
Personal identity is (in general) a wrong epistemic question asked by our moral intuition. Only if preference is expressed in terms of the territory (or rather in a form flexible enough to follow all possible developments), including the parts currently represented in moral intuition in terms of the-me-in-the-future object in the territory, will the confusion with expectations and anthropic thought experiments go away.