Probabilistic inference for general belief networks is NP-hard (see The Computational Complexity of Probabilistic Inference Using Bayesian Belief Networks (PDF)). Thus straitforward approach is not an option. The problem is more like finding computationally tractable yet sufficiently powerful subtype of belief networks.

I generally agree with paulfchristiano here. Regarding Q2, Q5 and Q6 I'll note that that aside from Nils Nilsson, the researchers in question do not appear to be familiar with the most serious existential risk from AGI: the one discussed in Omohundro's The Basic AI Drives. Researchers without this background context are unlikely to deliver informative answers on Q2, Q5 and Q6.

It's also that if you take things that improve one side of mental performance it's likely to harm another. This isn't massively surprising to me: you'd expect that if upping a single hormone level or whatever would simply improve performance overall then evolution would have 'found' it. But presumably the same is true of giving performance-enhancing drugs to less intelligent animals - or, for that matter, giving people steroids etc. to increase their physical performance.

But just because drugs to make you run faster might lower your life expectancy, that doesn't mean our current running speed is the best evolution or technology can achieve. The problem is that any complex adaptation, like intelligence, is going to be a 'sweet spot' in the sense that a random massive change in a single factor will make it less succesful. That doesn't mean that evolution, or potentially much more sophisticated technological enhancement, can't improve matters.

Also, the 'something's going to get worse' principle only holds if what we consider bad is the same as what evolution selects against. It could in principle be true that humans became much more intelligent if they lost something that made them capable of defending themselves, reproducing, making allies or whatever. If our aims are different to what benefits our genes' survival, we may well be able to improve on nature: as we do with artificial sweetners, sex with condoms and other cunning tricks.

Actually, a bit of all three. The one you can control the most is probably "dork", which unpacks as "someone with complex ideas who is too impatient/show-offy to explain their idiosyncratic jargon".

I'm a native English speaker, and I know that I still frequently sound "dorky" in that sense when I try to be too succinct.

It wasn't me, but I suspect the poor grammar didn't help. It makes it hard to understand what you were getting at.

Also, interesting thing happens if by the whim of the creator computer is given a goal of tiling universe with most common still life in it and universe is possibly infinite. It can be expected, that computer will send slower than light "investigation front" for counting encountered still life. Meanwhile it will have more and more space to put into prediction of possible treats for its mission. If it is sufficiently advanced, then it will notice possibility of existence of another agents, and that will naturally lead it to simulating possible interactions with non-still life, and to the idea that it can be deceived into believing that its "investigation front" reached borders of universe. Etc...

Too smart to optimize.

Problem 2 by Bayes rule.

N is a random variable (RV) of number of filled envelopes.

C is a RV of selected envelope contains coin. P(C) means P(C=true) when appropriate.

Prior distribution

P(N=n) = 1/(m+1)

by the problem setup

P(C|N=n) = n/m

by the rule of total probability

P(C)=sum_n P(C|N=n)P(N=n) = sum_n (n/m/(m+1))=m(m+1)/2/m/(m+1)=1/2

by Bayes rule

P(N=n|C) = P(C|N=n)P(N=n)/P(C) = 2n/m/(m+1)

Let C' is a RV of picking filled envelope second time.

by the problem statement

P(C'|N=n,C) = (n-1)/m

by the rule of total probability

P(C'|C)=sum_n P(C'|N=n,C)P(N=n|C) = ... substitutions and simplifications ... = 2(m-1)/(3m)

solving P(C'|C)=P(C) obtains

m=4

That's a really clear description! Thanks for summarizing it.

I suspect it's highly relevant that if someone were to actually grow up in a grayscale environment, they wouldn't be capable of experiencing blue. Even if the optic nerve had somehow retained the ability to transmit data from cones, the brain simply would not be wired for blue-processing. I'm pretty sure her brain would interpret a colored world the way a black-and-white television would. (This is my understanding of neuroscience, by the way, not my stab at philosophy.)

I haven't taken the time to think carefully about the implications of this. It just seems suspicious to me that one of the clearest descriptions of qualia I've encountered involves a process that's neurologically implausible to enact.

I don't understand what the question is getting at. You're right that I don't think about cones when I check which color a light is, but this is the mechanism by which it enters my brain: since different lights enter my brain in different ways it is no surprise I can differentiate between them.

The cones in the eye detect three different aspects of light (redness, greenness, blueness) and these are sent to the brain in three different fibers. By this mechanism we see there's nothing magic going on in telling the difference between two colors. I guess the rods (which detect variation in brightness) are more relevant to the question of which light is on though.