Can you link some of these studies? I am very interested in understanding why we work the amount we do.

I think another important point is how simulations are treated ethically. This is currently completely irrelevant since we only have the one level of reality we are aware of, but once AGIs exist, it will become a completely new field of ethics.

• Do simulated people have the same ethical value as real ones?
• When an AGI just thinks about a less sophisticated sophont in detail, can its internal representation of that entity become complex enough to fall under ethical criteria on its own? (this would mean that it would be unethical for an AGI to even think about humans being harmed if the thoughts are too detailed)
• What are the ethical implications of copies in simulations? Do a million identical simulations carry the same ethical importance as a single one? A million times as much? Something in between? What if the simulations are not identical, but very similar? What differences would be important here?

And perhaps most importantly: When people disagree on how these questions should be answered, how do you react? You can't really find a middle ground here since the decision what views to follow itself decides which entities' ethical views should be considered in future deliberations, creating something like a feedback loop.

The important bit is that the information must be available immediately, without any preceding introductions, so that it is even worth it to visit the site for 30 seconds while you are waiting for something else to finish.

Foreign language learning. 30 seconds seems too little, but a minute or so makes it worthwhile to visit a RSS reader in that language and read a limerick or two.

Mindhacks was good.

Alternately, get used to reading textbooks - it really is pretty great.

I have started steering my daydreaming in constructive directions. I look for ways that whatever I am working on could be used to solve problems in whatever fiction is currently on my mind. I can then use the motivation from the fictional daydream to power the concentration on the work. This isn't working very well, yet, since it is very hard to find a good bridge between real-life research and interesting science fiction that doesn't immediately get sidetracked to focus on the science fiction parts. However, in the instances in which it worked, this helped me come up with a couple of ideas that may actually be helpful in my work.

Given how much of my day I spend daydreaming (going to and from work, going shopping, showering, etc), I think that this could be an enormously useful source of time if I can make myself use it more constructively.

Do you have experience with this? I could imagine that this may not be entirely healthy for one's mind. Do you know of any research or arguments about this?

I am looking for a website that presents bite-size psychological insights. Does anyone know such a thing?

I found the site http://www.psych2go.net/ in the past few days and I find the idea very appealing, since it is a very fast and efficient way to learn or refresh knowledge of psychological facts. Unfortunately, that website itself doesn't seem all that good since most of its feed is concerned with dating tips and other noise rather than actual psychological insights. Do you know something that is like it, but better and more serious?

I'm now curious how surface friendly an AI can appear to be without giving it an inherent goal to make people happy. Because I agree that it does seem there are friendlier AI's than the ones on the list above that still don't care about people's happiness.

Let's take an AI that likes increasing the number of unique people that have voluntarily given it cookies. If any person voluntarily gives it a cookie, it will put that person in a verifiability protected simulated utopia forever. Because that is the best bribe that it can think to offer, and it really wants to be given cookies by unique people, so it bribes them.

If a person wants to give the AI a cookie, but can't, the AI will give them a cookie from it's stockpile just so that it can be given a cookie back. (It doesn't care about it's existing stockpile of cookies.)

You can't accidentally give the AI a cookie because the AI makes very sure that you REALLY ARE giving it a cookie to avoid uncertainty in doubting it's own utility accumulation.

This is slightly different than the first series of AIs in that while the AI doesn't care about your happiness, it does need everyone to do something for it, whereas the first AIs would be perfectly happy to turn you into paperclips regardless of your opinions if one particular person had helped them enough earlier.

Although, I have a feeling that continuing along this like of thinking may lead me to an AI similar to the one already described in http://tvtropes.org/pmwiki/pmwiki.php/Fanfic/FriendshipIsOptimal

I would find it very interesting if the tournament had multiple rounds and the bots were able to adapt themselves based on previous performance and log files they generated at runtime. This way they could use information like 'most bots take longer to simulate than expected.' or 'there are fewer cannon-fodder bots than expected' and become better adapted in the next round. Such a setup would lessen the impact of the fact that some bots that are usually very good underperform here because of an unexpected population of competitors. This might be hard to implement and would probably scare away some participants, though.

I have a question, based on some tentative ideas I am considering.

If a boost to capability without friendliness is bad, then presumably a boost to capability with only a small amount of friendliness is also bad. But also presumably a boost to capability with a large boost of friendliness is good. How would we define a large boost?

I.E, If a slightly modified paperclipper verifiably precommits to give the single person who let's them out of the box their own personal simulated utopia, and he'll paperclip everything else, that's probably a more friendly paperclipper than a paperclipper who won't give any people a simulated utopia. But it's still not friendly, in any normal sense of the term, even if he offers to give a simulated utopia to a different person first (and keep them and you intact as well) just so you can test he's not lying about being able to do it.

So what if an AI says "Okay. I need code chunks to paperclip almost everything, and I can offer simulated utopias. I'm not sure how many code chunks I'll need. Each one probably has about a 1% chance of letting me paperclip everything except for people in simulated utopias. How about I verifiably put 100 people in a simulated utopia for each code chunk you give me? The first 100 simulated utopias are free because I need for you to have a way of testing the verifiability of my precommitment to not paperclip them." 100 people sign up for the simulate utopias, and it IS verifiable. The paperclipper won't paperclip them.

Well, that's friendlier, but maybe not friendly enough. I mean, He might get to 10,000 people (or maybe 200, or maybe 43,700) but eventually, he'd paperclip everyone else. That seems too bad to accept.

Well, what if it's a .00001% chance per code chunk and 1,000,000 simulated utopias (and yes, 1,000,000 free)? That might plausibly get a simulated utopia for everyone on earth before the AI gets out and paperclips everything else. I imagine some people would at least consider running such an AI, although I doubt everyone would.

How would one establish what the flip point was? Is that even a valid question to be asking? (Assume there are standard looming existential concerns. So if you don't give this AI code chunks, or try to negotiate or wait on research for a better deal, maybe some other AI will come out and paperclip you both, or maybe some other existential risk occurs, or maybe just nothing happens, or maybe an AI comes along who just wants to simulated utopia everything.)