Mostly, "provably secure" means all layers, unless there's a fair bit of fine print with it.

I don't think that's possible. Then "provably secure" would have to include a proof that our model of physics is correct and complete.

More generally, a "proof" is something done within a strictly-defined logic system. By definition it makes assumptions, and proves something given those assumptions.

So we can conclude that although David was smart enough to escape his sandbox, he isn't yet at the level of understanding human-style humor.

I think we need better definition of problem we like to study here. Probably beliefs and values are not so undistinguishable

From this page ->

Human values are, for example:

• civility, respect, consideration;
• honesty, fairness, loyalty, sharing, solidarity;
• openness, listening, welcoming, acceptance, recognition, appreciation;
• brotherhood, friendship, empathy, compassion, love.

1. I think none of them we could call belief.

2. If these will define vectors of virtual space of moral values then I am not sure if AI could occupy much bigger space than humans do. (how much selfish or unwelcome or dishonest could AI or human be?)

3. On the contrary - because we are selfish (is it our moral value which we try to analyze?) we want that AI will be more open, more listening, more honest, more friend (etc) than we want or plan to be. Or at least we are now. (so are we really want that AI will be like us?)

4. I see the question about optimal level of these values. For example would we like to see agent who will be maximal honest, welcoming and sharing to anybody? (AI at your house which welcome thieves and tell them what they ask and share all?)

And last but not least - if we will have more AI agents then some kind of selfishness and laziness could help. For example to prevent to create singleton or fanatical mob of these agents. In evolution of humankind, selfishness and laziness could help human groups to survive. And lazy paperclip maximizer could save humankind.

We need good mathematical model of laziness, selfishness, openness, brotherhood, friendship, etc. We have hard philosophical tasks with deadline. (singularity is coming and dead in word deadline could be very real)

In the space of all possible values, human values have occupied a very small space, with the main change being who gets counted as moral agent (the consequences of small moral changes can be huge, but the changes themselves don't seem large in an absolute sense).

Or, if you prefer, I think it's possible the AI moral value changes will range so widely, that human value can essentially be seen as static in comparison.

Corrected, thanks!

I think your last line was meant to be "beliefs and values" rather than "preferences and values".

And I don't know it's a question of distinguishing beliefs from values, more of a question of whether values are stable. I personally don't think most individuals have a CEV, and even if many do, there's no reason to suspect that any group has one. This is especially true for the undefined group "humanity", which usually includes some projections of not-yet-existent members.

What sort of therapy would work for me? Ruminating is probably the main cause of it. Now that I've refuted my current fears, I find that I can't wrench the quantum world out of my head. Everything I feel is now tainted by DT.

You are talking about rationality and about fear. Your protocol could have several independent layers. You seems to think that your ideas produce your fear, but it could be also opposite. Your fear could produce your ideas (and it is definitely very probable that fear has impact on your ideas (at least on contents)). So you could analyze rational questions on lesswrong and independently solve your irrational part (=fear etc) with terapeuts. There could be physical or chemical reasons why you are concerning more than other people. Your protocol for dangerous ideas needs not only discussing it but also solve your emotional responses. If you like to sleep well then it could depend more on your emotional stability than on rational knowledge.

Wheelbarrows are useful even if all you have is short mostly-level paths, even if you don't have paths much longer than the width of a construction site. Then once those are in use, the incentive (and the ability) to lengthen and flatten other paths is greatly increased.

Wooded parts of the Americas did have some famously long paths though I don't know how passable they would be for carts.

@Nozick: we are plugged to machine (Internet) and virtual realities (movies, games). Do we think that it is wrong? Probably it is question about level of connection to reality?

@Häggström: there is contradiction in definition what is better. F1 is better than F because it has more to strive and F2 is better than F1 because it has less to strive.

@CEV: time is only one dimension in space of conditions which could affect our decisions. Human cultures are choosing cannibalism in some situations. SAI could see several possible future decisions depending on surroundings and we have to think very carefully which conditions are acceptable and which are not. Or we could choose what we choose in some special scene prepared for humanity by SAI.