So what do you expect to be the signs of arbitrary general reasoning? Humans run out of memory eventually. If a dolphin or a chimp can do arbitrary reasoning but lacks the capacity to keep long-chains inside but for this, what would you expect to see. I'm still not sure what actual testable distinction would occur in these cases, although in so far as I can think of what might arguably be evidence, it looks like dolphins pass, as you can see in this article already linked to in this thread.

The divide isn't absolute, I'm sure - I believe essentially all mammals have quite a bit of self-awareness, but only in humans does that facility seem to be good enough to allow the development of a chain of thought.

If dolphins or chimps did or did not have chains of thought how would be able to tell the difference?

What is the essential difference between human and animal intelligence? I don't actually think it's just a matter of degree. To put it simply, most brains are once-through machines. They take input from the senses, process it in conjunction with memories, and turn that into actions, and perhaps new memories. Their brains have lots of special-purpose optimizations for many things, and a surprising amount can be achieved like this. The brains are once-through largely because that's the fastest approach, and speed is important for many things. Human brains are still mostly once-through.

But we humans have one extra trick, which is to do with self-awareness. We can to an extent sense the output of our brains, and that output then becomes new input. This in turn leads to new output which can become input again. This apparently simple capability - forming a loop - is all that's needed to form a Turing-complete machine out of the specialized animal brain.

Without such a loop, an animal may know many things, but it will not know that it knows them. Because it isn't able to sense explicitly about it was just thinking about, it can't then start off a new thought based on the contents of the previous one.

The divide isn't absolute, I'm sure - I believe essentially all mammals have quite a bit of self-awareness, but only in humans does that facility seem to be good enough to allow the development of a chain of thought. And that small difference makes all the difference in the world.

Evolution, as an algorithm, is very much better as an optimizer of an existing design than it is as a creator of a new design. Optimizing the size of the brain of a creature is, for evolution, an easy problem. Making a better, more efficient brain is a much harder problem, and happens slowly, comparatively speaking.

The optimization problem is essentially a kind of budgeting problem. If I have a budget of X calories per day, I can spend it on X kilos of muscle, or Y grams of brain tissue. Both will cost me the same amount of calories, and each brings its own advantages. Since evolution is good at this kind of problem, we can expect that it will correctly find the point of tradeoff - the point where the rate of gain of advantage for additional expenditure on ANY organ in the body is exactly the same.

Putting it differently, a cow design could trade larger brain for smaller muscles, or larger muscles for smaller brain. The actual cow is found at the point where those tradeoffs are pretty much balanced.

A whale has a large brain, but it's quite small in comparison to the whale as a whole. If a whale were to double the size of its brain, it wouldn't make a huge dent in the overall calorie budget. However, evolution's balance of the whale body suggests that it wouldn't be worth it. Making a whale brain that much bigger wouldn't make the whale sufficiently better for it to cost in.

Where this argument basically leads is to turn the conventional wisdom on its head. People say that big brains are better because they are bigger. However, the argument that evolution can balance the size of body structures efficiently and quickly leads to the opposite conclusion. Modern brains are bigger because they are better. Because modern brains are better than they used to be - because evolution has managed to create better brains - it becomes more worthwhile making them bigger. Because brains are better, adding more brain gives you a bigger benefit, so the tradeoff point moves towards larger brain sizes.

Dinosaur brains were very much smaller, on the whole, than the brains of similar animals today. We can infer from this argument that this because their brains were less effective, and that in turn lowered any advantage that might have been gained from making the size of the brain larger. Consequently, dinosaurs must have been even more stupid than the small size of their brains suggests.

Although there is a nutritional argument for bigger brains in humans - the taming of fire allowed for much more efficient food usage - perhaps there is also some sense in which the human brain has recently become better, which in turn led it to become larger. Speculative, perhaps. But on the larger scale, looking at the sweeping increase in brain sizes across the whole of the geological record, the qualitative improvement in brains has to be seen in the gradual increase in size.

I think the interesting question is why we care for our future selves at all.

As kids, we tend not to. It's almost a standard that a child has a holiday, and a bit of homework to do during that holiday, then they will decide not to do the work at the beginning of the break. The reason is they care about their current selves, and not about their future self. Of course in due time the future becomes the present, and that same child has to spend the entire time at the end of their holiday working furiously on everything that's been left to the last minute. At that point, they wish that their past self had chosen an alternative plan. This is still not really wisdom, as they don't much care about their past self either - they care about their present self who now has to do the homework.

Summarising - if your utility function changes over time, then you will, as you mentioned, have conflict between your current and future self. This prevents your plans for the future from being stable - a plan that maximises utility when considered at one point no longer maximises it when considered again later. You cannot plan properly - and this undermines the very point of planning. (You may plan to diet tomorrow, but when tomorrow comes, dieting no longer seems the right answer....)

I think this is why the long view becomes the rational view - if you weight future benefits equally to your present ones, assuming (as you should) that your reward function is stable, then a plan you make now will still be valid in the future.

In fact the mathematical form that works is any kind of exponential - it's OK to have the past be more important than the future, or the future more important than the past as long as this happens as an exponential function of time. Then as you pass through time, the actual sizes of the allocated rewards change, but the relative sizes remain the same, and planning should be stable. In practice an exponential rise pushes all the importance of reward far out into the indefinite future, and is useless for planning. Exponential decays push all the important rewards into your past, but since you can't actually change that, it's almost workable. But the effect of it is that you plan to maximise your immediate reward to the neglect of the future, and since when you reach the future you don't actually think that it was worthwhile that your past self enjoyed these benefits at the expense of your present self, this doesn't really work either as a means of having coherent plans.

That leaves the flat case. But this is a learned fact, not an instinctive one.

More generally, Eliezer takes an explicit and very detail oriented approach to some aspects of some topics, but takes a very informal and relaxed approach to other topics. It seems to me that he pays attention to detail more when he agrees with the arguments he is outlining. This has the effect of making the arguments he opposes seem artificially weaker.

This isn't directly relevant to the chocolate cake issue, but your comment reminded me of this.

Koan 3:

Does the idea that everything is made of causes and effects meaningfully constrain experience? Can you coherently say how reality might look, if our universe did not have the kind of structure that appears in a causal model?

It's meaningful and false, rather than meaningless, to say that on March 22nd, 2003, the particles in the center of the Sun spontaneously arranged themselves into a short-lived chocolate cake. This statement's truth or falsity has no consequences we'll ever be able to test experientally. Nonetheless, it legitimately describes a way reality could be, but isn't; the atoms in our universe could've been arranged like that on March 22nd 2003, but they weren't.

I actually think this a confusing statement. From a thermodynamic perspective, it's not impossible that the particles in the center of the Sun spontaneously arranged themselves into a short-lived chocolate cake on that day. It's very, very, extremely unlikely, but not actually completely impossible.

The extreme unlikelihood (roughly equal to me temporarily becoming a chocolate cake myself) is such that we are justified, in terms of the approximation that is plain English, in saying that it is impossible that such a thing occurred, and that it is just wrong to claim that it happened. But this is using the usual rule of thumb that absolute truth and falsity isn't something we can actually have, so we happily settle for saying something is true or false when we're merely extremely sure rather than in possession of absolute proof.

It's quite OK in that context to claim that it's meaningless and false to claim that the chocolate cake appeared, as the claimant has no good reason to make the claim, and saying the claim is false is pointing out the lack of that reason. The bit I don't agree with is your final sentence.

Nonetheless, it legitimately describes a way reality could be, but isn't; the atoms in our universe could've been arranged like that on March 22nd 2003, but they weren't.

Here's where it gets confusing. If you are speaking in colloquial English, it's true to say that it's impossible that a chocolate cake could appear in the middle of the Sun, and therefore it didn't happen. If you're speaking more scientifically, it's instead true to say that it's possible that the atoms in the Sun's core could spontaneously form a chocolate cake, but the likelihood is of the order of 10^10^23 (or something like that) against, which clearly is sufficiently close to impossible for us to say informally that it didn't happen. As the sentence stands, you end up making a claim of knowledge which you don't have - that it was possible that a certain state of affairs could occur in the Sun, but that you know somehow that it didn't.

There will always be multiple centers of power

is not a well supported statement. Two main trends argue against this

1 with technological gaps the number of center is th number of agents with tech

In lots of colonies there were many centers of power, then colonists with guns landed and there was 1 center. Post ww2 the us had the bomb and strategic superiority. The soviets matched tech and there were two power centers. When Britan ruled the seas there was one major naval power In many developing nations in the cold war there were 2 powers centers that corresponded to us and soviet supplied arms.

It's not crazy to think AI or whatever technology it can invent as being another instance of tech superiority reducing the number of power centers

In this case my judgement of the probabilities is that we are completely justified in ignoring the threat.

Do you consider my pascals mugging to be less likely then the general examples of the genre, or do you think that all pascals muggings" probabilities are that we are completely justified in ignoring the threat."