I think the correct objection is something you can't quite see in google maps. If you program an AI to do nothing but output directions, it will do nothing but output directions. If those directions are for driving, you're probably fine. If those directions are big and complicated plans for something important, that you follow without really understanding why you're doing (and this is where most of the benefits of working with an AGI will show up), then you could unknowingly take over the world using a sufficiently clever scheme.

Also note that it would be a lot easier for the AI to pull this off if you let it tell you how to improve its own design. If recursively self-improving AI blows other AI out of the water, then tool AI is probably not safe unless it is made ineffective.

This does actually seem like it would raise the bar of intelligence needed to take over the world somewhat. It is unclear how much. The topic seems to me to be worthy of further study/discussion, but not (at least not obviously) a threat to the core of SIAI's mission.

I don't feel like my apathy is abnormal in and of itself, but combined with being more aware of the big picture, and thinking more about the future, it seems more troubling. Most people have nothing but the life in front of their noses, working at a grocery store or whatever, and so their listlessness is entirely natural. I'm different. Like you, and like many people on this site, I have vision, I can see that the world is at a crossroads and that I have the potential to change its course. And yet, I still feel nothing, while it seems like the rest of you are enthusiastic.

This is roughly how I feel as well.

It seems to me that it is more about a certain personality type than intelligence. I know highly intelligent people who would not ask the smartest or most knowledgeable person what to do because they damn well want to figure it out for themselves.

Barring communication usually works, though the cure is generally considered to be worse than the disease.

Yes. Because you're huffing priors. Twice as much, in fact - we have to make up one number, you have to make up two.

How do you choose the interval? I have not been able to see any method other than choosing something that sounds good (choosing the minimum and maximum conceivable would lead to silly Pascal's Wager - type things, and probably total paralysis.)

The discontinuity: Suppose you are asked to put a fair price f(N) on a bet that returns N if A occurs and 1 if it does not. The function f will have a sharp bend at 1, equivalent to a discontinuity in the derivative.

An alternative ambiguity aversion function, more complicated to define, would give a smooth bend.

Yes. So basically you are biting a certain bullet that most of us are unwilling to bite, of not having a procedure to determine your decisions and just kind of choosing a number in the middle of your range of choices that seems reasonable.

You're also biting a bullet where you have a certain kind of discontinuity in your preferences with very small bets, I think.

I don't get what this range signifies. There should be a data point about how ambiguous it is, which you could use or not use to influence actions. (For instance, if someone says they looekd in the urn and it seemed about even, that reduced ambiguity.) But then you want to convert that into a range, which does not refer to the actual range of frequencies (which could be 1/3 +- 1/3) and is dependent on your degree of aversion, but then you want to convert that into a decision?

There seems to be an issue of magnitude here. There are 3 possible ways the urn can be filled:

1. It could be selected uniformly at random
2. It could be selected through some unknown process: uniformly at random, biased against me, biased towards blue, biased towards green, always exactly 30/30, etc.
3. It could be selected so as to exactly minimize my profits

2 seems a lot more like 1 than it does like 3. Even without using any Bayesian reasoning, a range is a lot more like the middle of the range than it is like one end of the range.

(This argument seems to suggest a "common-sense human" position between high ambiguity aversion and no ambiguity aversion, but most of us would find that untenable.)

An alternative way of talking about it:

The point I am making is that it is much more clear which direction my new information is supposed to influence you then your information is supposed to influence me. If a variable x is in the range [0,1], finding out that it is actually 0 is very strongly biasing information. For instance, almost every value x could have been before is strictly higher than the new known value. But finding out that it is 1/2 does not have a clear direction of bias. Maybe it should make you switch to more confidently betting x is high, maybe it should make you switch to more confidently betting x is low. I don't know, it depends on details of the case, and is not very robust to slight changes in the situation.

In which direction should it change my behavior? What does it push me towards?