= 27d567bc2d48d9f40e9ccc20ea370b15)
I agree with the points about Boltzmann Brains and mind substrates. In those cases, though, I'm not sure the FAI heuristic saves you any work, compared to just directly asking what the right answer is.
The ideal FAI wouldn't care about its personal identity over time
Almost certainly not true if taken verbatim; one of the critical traits of an FAI (as opposed to a regular AGI) is that certain traits must remain stable under self-improvement. An FAI would care very strongly about certain kinds of changes. But with a less literal reading, I can see what you're going for here - yes, an ideal FAI might be indifferent to copying/deletion except to the extent that those help or hinder its goals.
I'm not sure how that belief, applied to oneself, cashes out to anything at all, at least not with current technology. I also don't see any reason to go from "the FAI doesn't care about identity" to "I shouldn't think identity exists."
The ideal FAI would use UDT/TDT/etc. Therefore I should too.
(Disclaimer: I am not a decision theorist. This part is especially likely to be nonsense.)
You should use which one?
The less snappy version is that TDT and UDT both have problem cases. We don't really know yet what an ideal decision theory looks like.
Second, I doubt any human can actually implement a formal decision theory all the time, and doing it only part-time could get you "valley of bad rationality"-type problems.
Third, I suspect you could easily run into problems like what you might get by saying "an ideal reasoner would use Solomonoff Induction, so I should too". That's a wonderful idea, except that even approximating it is computationally insane, and in practice you won't get to use any of the advantages that make Solomonoff Induction theoretically optimal.
If you instead mean things like "an ideal FAI would cooperate in PD-like scenarios given certain conditions", then sure. But again, I'm not sure the FAI heuristic is saving you any work.
The ideal FAI would ignore uncomputable possibilities. Therefore I should too.
A factual FAI might, for mere practical reasons. I don't see why an ideal FAI normatively should ignore them, though.
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
An "ideal" being is many layers of "better" than you are, whereas something that is simply better is only one layer better. To get to ideal, you would have to imagine someone better, then imagine what that person would consider better, and so forth, until you hit a state where there are no further improvements to be made.
In the picture you just drew, the ideal being is derived from a series of better beings, thus it is (trivially) easier to imagine a better being than to imagine an ideal being.
I see it differently: The ideal being maximizes all good qualities, whereas imperfect beings have differing levels of the various good qualities. Thus to compare a non-ideal being to an ideal being, we only need to recognize how the ideal being does better than the non-ideal being in each good quality. But to compare two non-ideal beings, we need to evaluate trade-offs between their various attributes (unless one is strictly greater than the other)
Thinking about it more, I am not happy with either of the above models. One question that arises is: Does the same reasoning extend to other cases as well? i.e. are we better off thinking about incremental improvements than about the ideal society? Are we better off thinking about incremental improvements than about the ideal chess algorithm?
I think in some cases maybe we are, but in some cases we aren't--ideals are useful sometimes. I'd go farther to say that some aspects of many ideals must be arrived at by iterating, but other aspects can be concluded more directly. An uninteresting conclusion, but one that supports my overall point: I wasn't claiming that I knew everything about the ideal FAI, just that I had justified high confidence in some things.