"If you are constantly surprised by solutions that are high in you preference ordering but low in your search ordering, that is a problem with your search ordering. If your search ordering is correct, creativity is useless."
... Yeah, and optimization is trivial, just use the correct search ordering.
simon: "Just for the sake of devil's advocacy: 4) You want to attribute good things to your ethics, and thus find a way to interpret events that enables you to do so."
Eliezer: "The universe isn't set up to reward virtue - so why did my ethics help so much? Am I only imagining the phenomenon? That's one possibility."
A sidenote:
Eliezer: "It has something to do with natural selection never choosing a single act of mercy, of grace, even when it would cost its purpose nothing: not auto-anesthetizing a wounded and dying gazelle, when its pain no longer serves even the adaptive purpose that first created pain."
It always costs something; it is cheaper to build a gazelle that always feels pain than one that does so until some conditions are met. This is the related to the case of supposing a spaceship that has passed out of your lightcone still exists.
Natural selection isn't fair; it doesn't compute fairness at all. This has nothing to do with the question whether there are situations where it could be fair to no cost, even if perhaps humans are more easily made aware of this fact if presented with cases where natural selection is outright unfair. Such cases very probably exists (ponder gene fixation) but to find them is nigh to impossible: you have to prove an absence of costs.
I think it is quite possible for rich individuals to create structures surviving themselves such that it would be very hard to distinguish them for institutes/foundations/whatever that has a living person at the bottom. I'm not very familiar with the subject, but I would guess that there exists accounts on the Cayman Islands and similar states whose owners have died but the owners is too well hidden for anyone to find out.
Concerning the academic interest, I think that coming generations will, like us, find the preceding documentation terribly lacking. "Written records and movies? Why didn't they upload and archive their brains?" People living in the early days of writing would probably consider themselves as saving all that anyone in later times could wish to know, and compared to a society without writing, they would be at least partly justified in that belief. We're in the middle of the fourth information revolution (the preceding being the invention of writing, book binding and the printing press*) and we shouldn't underestimate in what new ways and to what higher extends information will be stored and used in the future.
*You probably could add language and perhaps some other things I haven't thought of, it depends on what timescales you're interested in.
("In any case, in the event that radical life extension is already here, there's just no need to solve the problem of defrosting frozen brains for paying customers so I'd expect that to be, at least, put on the back burner." I guess you just where careless with the "In any case" here; in exactly the case we where discussing, the assertion is false.)
"in which case there will be nobody signing up for cryonics and no market to develop the technology to defrost and revive." ...except for all that has already died while being signed up. Their wills and possibly foundations they set up would provide large enough a market if only a sufficient amount of rich people do sign up.
Another point: if we just achieved the defrosting techniques and had a bunch of cryonically suspended individuals from say the 19th century, we would be ankle deep in the saliva from all psycologists, historians etc. that would spend all their funding to get a chance to talk to such an individual.
Disclaimer: I am not signed up for cryonics. It's just that I think that some of the dismissals offered here is not well thought through.
"Other than modeling experiments, it's hard to even test the theory because the object of study would be scientific communities and it would be difficult and (ahem!) ethically dubious to experiment on them..."
"You used every avenue available to you, in seeking knowledge; that was respected here."
Heh.
"I disagree; I think the underspecification is a more serious issue than the uncomputability. There are constant factors that outweigh, by a massive margin, all evidence ever collected by our species."
Agreed. The constant factors really are a problem. If one has taken a few information theory courses, it's easy to disregard it as one usually uses Kolmogorov on e.g. symbol sequences in the infinite limit. When comparing two theories though, they have finite size and thus constants does matter. It is probably possible to find two Turing machines such that two competing models have equal length on their respective best machines even if they differ greatly when tested on one of them.
It may be possible to construct an argument that favors an interpreter over all others, Sebastian Hagen gave a few ideas above, but it is highly non-trivial.
Wiseman: "It's not junk DNA, it merely has usefulness in many different configurations."
Perhaps it was unfortunate to use the term junk DNA. What I was thinking of was more on the lines of information content. If a given base is useful in several configurations, it contains that much less information.
If base X has to be e.g. guanine to have its effect, that is one out of four possible states, i.e. 2 bits of information. If it could be either guanine or thymine, then it only contains one bit.
It may be that the actual human genome uses more than 5*10^7 bases to encode 10^8 bits of information. The total information would still have to fit in 25 MB after you got rid of redundancy because of the limitations in the information upholding abilities of evolution. (Assuming the rest of Eliezers calculations.)
Even if most mutations is neutral, that just says that most of the genome don't contain any information. If you flip a base and it doesn't make any difference, then you've just proved that it was junk-DNA, right?
The following may or may not be relevant to the point Unknown is trying to make.
I know I could go out and kill a whole lot of people if I really wanted to. I also know, with an assigned probability higher than many things I consider sure, that I will never do this.
There is no contradiction between considering certain actions within the class of things you could do (your domain of free will if you wish) and at the same time assign a practically zero probability to choosing to take them.
I envision a FAI reacting to a proof of its own friendliness with something along the lines of "tell me something I didn't know".
(Do keep in mind that there is no qualitative difference between the cases above; not even a mathematical proof can push a probability to 1. There is always room for mistakes.)