only that they would be VNM-rational

But if the agent can't be subject to Dutch books, what's the point of being VNM-rational? (in fact, in my construction, the agent need not be initially complete).

But the main point is that VNM-rational isn't clearly defined. Is it over all possible decisions, or just over decisions the agent actually faces? Given that rationality is often defined on Less Wrong in a very practical way (generalised "winning") I see no reason to need to assume the first. It weakens the arguments for VNM-rationality, makes it into a philosophical ideal rather than a practical tool.

And so while it's clear that an AI would want to make itself into an unlosing agent, it's less clear that it would want to make itself into an expected utility maximiser. In fact, it's very clear that in some cases it wouldn't: if it knew that outcomes A and B were impossible, and it currently didn't have preferences between them, then there is no reason it would ever bother to develop preferences there (baring social signalling and similar).

I've just made an enrollment deposit at the University of Illinois at Urbana-Champaign, and I'm wondering if any other rationalists are going, and if so, would they be interested in sharing a dorm?

"What is the experience of the other people I'm interacting with?"

I have sometimes found empathy to be a complicated concept, but this question really cuts to the heart of it and causes your brain to model the situation from the other person's perspective and use that in your decision-making.

Intriguingly, this seems to even apply to interactions with my future selves. If I don't ask this question or one like it, I'm likely to write a massive todo list for myself—probably a completely impossible list, and delegate it to my tomorrow!self with very little thought. Then when that self encounters the list, it's so overwhelming and inconsiderate that I find it hard to deal with at all. If instead, I stop to wonder what the experience will be of my future self encountering the tasks I've delegated for it, I realize that it makes sense to prioritize much of that up front, and to frame the delegation process with more context around why my past self wanted the thing done. This is basically the difference between saying "Future self, do this" and "Future self, at the moment it seems to make sense to me that you do this stuff, for reasons X, Y, Z." or some similarly empathetic request... which is much less likely to produce e.g. reactance.

It's easier to think about unpredictability without picturing Many Worlds - e.g. do we say "don't worry about driving too fast because there will be plenty of worlds where we don't kill anybody?"

Yes, the problem is that it is easy to imagine Many Worlds... incorrectly.

We care about the ratio of branches where we survive, and yet, starting with Big Bang, the ratio of branches where we ever existed is almost zero. So, uhm, why exactly should we be okay about this almost zero, but be very careful about not making it even smaller? But this is what we do (before we start imagining Many Worlds).

So for proper thinking perhaps it is better to go with collapse intepretation. (Until someone starts making incorrect conclusions about mysterious properties of randomness, in which case it is better to think about Many Worlds for a moment.)

I would immediately download this iff it had a GUI.

My understanding was that this was about whether the singularity was "AI going beyond "following its programming"," with goal-modification being an example of how an AI might go beyond its programming.

Hem hem.

http://lesswrong.com/lw/ic/the_virtue_of_narrowness/

There's a difference, I think. It's just that we havn't quite grasped it yet. I had it until I read your post, and then I lost it. It's like in Harry Potter and the Methods of Rationality when Harry learns Partial Transfiguration.

In this Harry Potter universe, you can you magic to change things into other things, but you can't change only part of a thing into another thing, for example you can change a wall into marshmallow in order to escape a room, but you would have to dispense the energy of changing the entire wall, and not just make yourself a little marshmallow hole.

Well Harry learns to violate this rule with science, because nothing in the world is really connected, it's just an illusion in our heads. So Harry can now transfigure only part of a rubber eraser into steel, if he wants to. This is all making perfect sense, right?

But Proffesor MacGonagall is skeptical:

"Harry's idea stemmed from simple ignorance, nothing more. If you changed half of a metal ball into glass, the whole ball had a different Form. To change the part was to change the whole, and that meant removing the whole Form and replacing it with a different one. What would it even mean to Transfigure only half of a metal ball? That the metal ball as a whole had the same Form as before, but half that ball now had a different Form?"

See, that makes sense too. And now everyone is confused. But partial transfiguration does exist (well, in the story) and the difference is that Harry could change a spot on the metal ball to glass in five minutes, instead of the thirty minutes it would have taken him to change the entire metal ball into a metal ball with a glass spot.

There's probably a distinction between laws with exceptions and new laws that you and I just don't know about yet.

Anyone care to enlighten us?

The thing is that PTSD is really not that binary, like many mental illnesses, it has a wide range of symptoms and severity levels. What Nancy is talking about is how one death can push one drastically over, skipping much of the middle range where it might be ambiguous if one had symptoms severe enough to be diagnoseable. (Disclaimer, while I've heard the same sort of things NancyLebovitz is talking about, I'm not aware of any studies actually supporting this.)

I've read the same thing about Nazi soldiers, and also that they couldn't handle another early method of killing-- driving prisoners around in closed trucks with the exhaust fed into the back compartment.

It's not that the thousands have no impact, it's that one person can make a much larger emotional difference.

I've also heard that for soldiers, seeing one more death or injury can be the tipping point into PTSD.