Perhaps his server is underspecced? It's currently slowed to an absolute c r a w l. What little I have seen certainly looks worthwhile, though.

I like this idea, but am seriously concerned about its effect on eye health. Weak eye muscles are not a thing you want to have, even if you live in the safest place in the world.

I don't see how understanding, acceptance, and love follow from rationality.

They do not follow from it, they are necessary to it.

• You need to relate well to yourself and others (love) in order to actually accomplish anything worthwhile without then turning around and sabotaging it.
• If you discover something, you need to accept what is actually going on in order to come to understand it, and understand it in order to apply it.

Are you saying that rationalism is a "philosophy of life", even leaving the soundness aside for a minute?

No. But a story that is trying to have broad appeal needs these things, whether it's a story about rationality or about watching paint dry. A story conveys a sense of life.

The parent comment said: "You need a good story. That's all. A good story."

That's not vague. That's wrong.

That depends on what you think 'good' is supposed to imply there. If 'convincing' is the intended connotation, then yeah, wrong. If 'consistent' is the intended connotation, that is not obviously wrong, People need stories to help them get stuff done, even though stories are overall pretty terrible.

Science, for example, has methods, but overall science is a story about how to get accurate data and interpret it accurately in spite of our human failings. The way that the elements of that story were obtained does not make it any less of a story. History itself is a story, no matter how accurate you get, it remains a narrative rather than a fact. Reality exists, but all descriptions of it are stories; there are no facts found in stories; Facts are made of reality, not of words.

Belittling their sense of the distinction seems like a pretty unfriendly thing to do.

However, some of the things that religion proscribes are also pretty unfriendly things to do. (arguably religion itself is an unfriendly thing to do). So until they stop doing such things, they can reasonably expect to cop retaliation for their own hostility. (in proportion to their craziness, IME -- for example Buddhism doesn't cop much flak, whereas Christianity does)

Putting yourself in the position of being significantly opposed to reality cannot be rationally viewed as a friendly, or even just not-unfriendly, thing to do... You just never can constrain the effect to yourself. You tend to end up doing things that it is interesting for a character to do, but destructive in real life. And all in pursuit of mere 'good feelings'.

But this is not a specific problem of rationalists.. It's a broader problem with Western culture. Feeling strongly about things is not 'cool' or impressive. Plenty of people enjoy complaining, but passion makes you an alien.. perhaps an inspiring one, but ultimately an alien.. a person of whom people say 'oh, I could never do that', even if in the other breath they praise your passion and dedication.

I hesitate to assign a definite cause for that, but I am willing to comment that Western society somewhat deifies disaffected sociopathy, through its presentations in media, and also that I have a strong impression that most people have encountered enflamed evangelists, and they don't want to be that person. Whether they recognize it as ugly or not, they treat it as ugly, though spectacular.

I find I can always count on you to make pointlessly snarky comments.

I would prefer to be more specific, and say 'understanding, acceptance, confidence, control, and love' (with clear definitions for each, probably similar to the ones in the GROW Blue Book). Not all of these things can be used to make clever, snappy remarks to wow outsiders, but they are all necessary for a satisfying life, and therefore must be addressed effectively by any sound philosophy of life. The parent comment was only vague, not wrong.

That would be equivalent to self-sabotage or an attempt to systematically deny that 'you' possess some particular attribute A (eg. homosexuality, atheism..) which you do in fact possess, so.. no.

You are allowed to try to talk me into murdering someone, e.g. by appealing to facts I do not know; or pointing out that I have other preferences at odds with that one, and challenging me to resolve them; or trying to present me with novel moral arguments. You are not allowed to hum a tune in such a way as to predictably cause a buffer overflow that overwrites the encoding of that preference elsewhere in my cortex

.. And?

Don't you realize that this is just like word laddering? Any sufficiently powerful and dedicated agent can convince you to change your preferences one at a time. All the self-consistency constraints in the world won't save you, because you are not perfectly consistent to start with, even if you are a digitally-optimized brain. No sufficiently large system is fully self-consistent, and every inconsistency is a lever. Brainwashing as you seem to conceive of it here, would be on the level of brute violence for an entity like CelestAI.. A very last resort.

No need to do that when you can achieve the same result in a civilized (or at least 'civilized') fashion. The journey to anywhere is made up of single steps, and those steps are not anything extraordinary, just a logical extension of the previous steps.

The only way to avoid that would be to specify consistency across a larger time span.. which has different problems (mainly that this means you are likely to be optimized in the opposite direction -- in the direction of staticness -- rather than optimized 'not at all' (i think you are aiming at this?) or optimized in the direction of measured change)

TLDR: There's not really a meaningful way to say 'hacking me is not allowed' to a higher level intelligence, because you have to define 'hacking' to a level of accuracy that is beyond your knowledge and may not even be completely specifiable even in theory. Anything less will simply cause the optimization to either stall completely or be rerouted through a different method, with the same end result. If you're happy with that, then ok -- but if the outcome is the same, I don't see how you could rationally favor one over the other.

It is possible for a Lars to exist, and prefer not to change anything about the way he lives his life, and prefer that he prefers that, in a coherent, self-endorsing structure, and there be nothing you can do about it.

It is, of course, the last point that I am contending here. I would not be contending it if I believed that it was possible to have something that was simultaneously remotely human and actually self-consistent. You can have Lars be one or the other, but not both, AFAICS.

Once she has the situation completely under control, however, she has no excuses left - absolute power is absolute responsibility.

This is the problem I'm trying to point out -- that the absolutely responsible choice for a FAI may in some cases consist of these actions we would consider unambiguously abusive coming from a human being. CelestAI is in a completely different class from humans in terms of what can motivate her actions. FAI researchers are in the position of having to work out what is appropriate for an intelligence that will be on a higher level from them. Saying 'no, never do X, no matter what' is not obviously the correct stance to adopt here, even though it does guard against a range of bad outcomes. There probably is no answer that is both obvious and correct.

I'm puzzled. I read you as claiming that your notion of 'strengthening people' ought to be applied even in a fictional situation where everyone involved prefers otherwise. That's kind of a moral claim.

In that case I miscommunicated. I meant to convey that if CelestAI was real, I would hold her to that standard, because the standards she is held to should necessarily be more stringent than a more flawed implementation of cognition like a human being. I guess that is a moral claim. It's certainly run by the part of my brain that tries to optimize things.

(And as for "animalisticness"... yes, technically you can use a word like that and still not be a moral realist, but seriously? You realise the connotations that are dripping off it, right?)

I mainly chose 'animalisticness' because I think that a FAI would probably model us much as we see animals -- largely bereft of intent or consistency, running off primitive instincts.

I do take your point that I am attempting to aesthetically optimize Lars, although I maintain that even if no-one else is inconvenienced in the slightest, he himself is lessened by maintaining preferences that result in his systematic isolation.

Gandhi does not prefer to murder. He prefers to not-murder. His human brain contains the wiring to implement "frothing lunacy", sure, and a little pill might bring it out, but a pill is not a fact. It's not even an argument.

No pills required. People are not 100% conditionable, but they are highly situational in their behaviour. I'll stand by the idea that, for example, anyone who has ever fantasized about killing anyone can be situationally manipulated over time to consciously enjoy actual murder. Your subconscious doesn't seem to actually know the difference between imagination and reality, even if you do.

Perhaps Gandhi could not be manipulated in this way due to preexisting highly built up resistance to that specific act. If there is any part of him, at all, that enjoys violence, though, it's a question only of how long it will take to break that resistance down, not of whether it can be.

People do experience dramatic and beneficial preference reversals through experiencing things that, on the whole, they had dispreferred previously.

Yes, they do. And if I expected that an activity would cause a dramatic preference reversal, I wouldn't do it.

Of course. And that is my usual reaction, too, and probably even the standard reaction -- it's a good heuristic for avoiding derangement. But that doesn't mean that it is actually more optimal to not do the specified action. I want to prefer to modify myself in cases where said modification produces better outcomes. In these circumstances if it can be executed it should be. If I'm a FAI, I may have enough usable power over the situation to do something about this, for some or even many people, and it's not clear,as it would be for a human, that "I'm incapable of judging this correctly".

In case it's not already clear, I'm not a preference utilitarian -- I think preference satisfaction is too simple a criteria to actually achieve good outcomes. It's useful mainly as a baseline.

Huh? She's just changing people's plans by giving them chosen information, she's not performing surgery on > their values Did you notice that you just interpreted 'preference' as 'value'? This is not such a stretch, but they're not obviously equivalent either.

I'm not sure what 'surgery on values' would be. I'm certainly not talking about physically operating on anybody's mind, or changing that they like food, sex, power, intellectual or emotional stimulation of one kind or another, and sleep, by any direct chemical means, But how those values are fulfilled, and in what proportions, is a result of the person's own meaning-structure -- how they think of these things. Given time, that is manipulable. That's what CelestAI does.. it's the main thing she does when we see her in interactiion with Hofvarpnir employees.

In case it's not clarified by the above: I consider food, sex, power, sleep, and intellectual or emotional stimulation as values, 'preferences' (for example, liking to drink hot chocolate before you go to bed) as more concrete expressions/means to satisfy one or more basic values, and 'morals' as disguised preferences.

people like to Google-stalk everyone they come into contact with,

People do that?

People have too much time on their hands. Geez.