It seems to me that you should only do this if everyone has utility functions that are completely anthropically selfish (i.e. they only care about their own subjective experience). Otherwise, wouldn't it be cruel to intentionally simulate a world with so many unpleasant characteristics that we could otherwise remove if we weren't focused on making the simulation subjectively indistinguishable from our own world?
As such, I don't think we should commit to any such thing.
The point you raise is by far the strongest argument I know of against the idea.
However, it is a moral objection rather than a decision-theory objection. It sounds like you agree with me on the decision theory component of the idea: that if we were anthropically selfish, it would be rational for us to commit to making ancestor-simulations with afterlives. That's an interesting result in itself, isn't it? Let's go tell Ayn Rand.
When it comes to the morality of the idea, I might end up agreeing with you. We'll see. I think there are several minor considerations in favor of the proposal, and then this one massive consideration against it. Perhaps I'll make a post on it soon.
This is a formal version of a real-life problem I've been thinking about lately.
Should we commit to creating ancestor-simulations in the future, where those ancestor-simulations will be granted a pleasant afterlife upon what appears to their neighbors to be death? If we do, then arguably we increase the likelihood that we ourselves have a pleasant afterlife to look forward to.
The thing is, if you get suspicious you don't immediately leap to the conclusion of chatbots. Nobody glances around, realizes everyone is bland and stupid and thinks," I've been fooled! An AI has taken over the world and simulated chatbots as human beings!" unless they suffer from paranoia.
Your question, "How much more sophisticated would they need to be" is answered by the question "depends". If you live as a hermit in a cave up in the Himalayas, living off water from a mountain stream and eating nothing but what you hunt or gather with your bare hands, the AI will not need to use chatbots at all. If you're a social butterfly who regularly talks with some of the smartest people in the world, the AI will probably struggle(let's be frank; if we were introduced to a chatbot imitating Eliezer Yudkosky the difference would be fairly obvious).
If you've interacted a lot with your friends and family, and have never been once suspicious that they are a chatbot, then with our current level of AI technology it is unlikely(but not impossible) that they are actually chatbots.
[Please note that if everyone says what you expect them to say that would make it fairly obvious something is up, unless you happen to be a very, very good predictor of human behavior.]
Thanks for the response. Yes, it depends on how much interaction I have with human beings and on the kind of people I interact with. I'm mostly interested in my own case, of course, and I interact with a fair number of fairly diverse, fairly intelligent human beings on a regular basis.
If you're a social butterfly who regularly talks with some of the smartest people in the world, the AI will probably struggle
Ah, but would it? I'm not so sure, that's why I made this post.
Yes, if everyone always said what I predicted, things would be obvious, but recall I specified that random variation would be added. This appears to be how dream characters work: You can carry on sophisticated conversations with them, but (probably) they are governed by algorithms that feed off your own expectations. That being said, I now realize that the variation would have to be better than random in order to account for how e.g. EY consistently says things that are on-point and insightful despite being surprising to me.
Consider the Tatiana Maslany as Mara the deceiver strategy. There is already one true person in the system other than you. The AI. That means everyone you interact with can be arbitrarily sophisticated without adding any more people to the system - All it takes is for the AI to directly puppet them. Acting in place of making bots.
This is, however, not a fruitful line of thinking, If the universe were really out to get you to this degree you would be better of being oblivious.
Yes, but I'm not sure there is a difference between an AI directly puppeting them, and an AI designing a chatbot to run as a subroutine to puppet them, at least if the AI is willing to monitor the chatbot and change it as necessary. Do you think there is?
Also, it totally is a fruitful line of thinking. It is better to believe the awful truth than a horrible lie. At least according to my values. Besides, we haven't yet established that the truth would be awful in this case.
Define "make them people". Because it sounds like what you're talking about is just the problem of other minds with "chatbot" substituted for "zombie"
I'm surprised that it sounded that way to you. I've amended my original post to clarify.
The amount of detail an AI would need for simulating realistic NPC's for you may be influenced substantially by a significant list of things like whether you are an introvert or an extrovert, what your job is, how many people you interact with over the course of a day and to what level of detail, and how many of those people you have very deep conversations with and how frequently, and if an acceptable AI answer to you mentioning to someone 'Everyone and everything seems so depressingly bland and repetitive' is a doctor telling you 'Have you tried taking medication? I'll write you a prescription for an antidepressant.'
Yes, this is the sort of consideration I had in mind. I'm glad the discussion is heading in this direction. Do you think the answer to my question hinges on those details though? I doubt it.
Perhaps if I was extraordinarily unsuspicious, chatbots of not much more sophistication than modern-day ones could convince me. But I think it is pretty clear that we will need more sophisticated chatbots to convince most people.
My question is, how much more sophisticated would they need to be? Specifically, would they need to be so much more sophisticated that they would be conscious on a comparable level to me, and/or would require comparable processing power to just simulating another person? For example, I've interacted a ton with my friends and family, and built up detailed mental models of their minds. Could they be chatbots/npcs, with minds that are nothing like the models I've made?
(Another idea: What if they are exactly like the models I've made? What if the chatbot works by detecting what I expect someone to say, and then having them say that, with a bit of random variation thrown in?)
How long is a piece of string?
If you did live in such a world where everything was based around you then the controller could allocate resources even more efficiently by monitoring your mind and devoting more computational power to making the people around you believable when you get suspicious.
That's exactly what I had in mind, although I did specify that the controller would never simulate anybody besides me to the level required to make them people.
An "ideal" being is many layers of "better" than you are, whereas something that is simply better is only one layer better. To get to ideal, you would have to imagine someone better, then imagine what that person would consider better, and so forth, until you hit a state where there are no further improvements to be made.
In the picture you just drew, the ideal being is derived from a series of better beings, thus it is (trivially) easier to imagine a better being than to imagine an ideal being.
I see it differently: The ideal being maximizes all good qualities, whereas imperfect beings have differing levels of the various good qualities. Thus to compare a non-ideal being to an ideal being, we only need to recognize how the ideal being does better than the non-ideal being in each good quality. But to compare two non-ideal beings, we need to evaluate trade-offs between their various attributes (unless one is strictly greater than the other)
Thinking about it more, I am not happy with either of the above models. One question that arises is: Does the same reasoning extend to other cases as well? i.e. are we better off thinking about incremental improvements than about the ideal society? Are we better off thinking about incremental improvements than about the ideal chess algorithm?
I think in some cases maybe we are, but in some cases we aren't--ideals are useful sometimes. I'd go farther to say that some aspects of many ideals must be arrived at by iterating, but other aspects can be concluded more directly. An uninteresting conclusion, but one that supports my overall point: I wasn't claiming that I knew everything about the ideal FAI, just that I had justified high confidence in some things.
I agree with the points about Boltzmann Brains and mind substrates. In those cases, though, I'm not sure the FAI heuristic saves you any work, compared to just directly asking what the right answer is.
The ideal FAI wouldn't care about its personal identity over time
Almost certainly not true if taken verbatim; one of the critical traits of an FAI (as opposed to a regular AGI) is that certain traits must remain stable under self-improvement. An FAI would care very strongly about certain kinds of changes. But with a less literal reading, I can see what you're going for here - yes, an ideal FAI might be indifferent to copying/deletion except to the extent that those help or hinder its goals.
I'm not sure how that belief, applied to oneself, cashes out to anything at all, at least not with current technology. I also don't see any reason to go from "the FAI doesn't care about identity" to "I shouldn't think identity exists."
The ideal FAI would use UDT/TDT/etc. Therefore I should too.
(Disclaimer: I am not a decision theorist. This part is especially likely to be nonsense.)
You should use which one?
The less snappy version is that TDT and UDT both have problem cases. We don't really know yet what an ideal decision theory looks like.
Second, I doubt any human can actually implement a formal decision theory all the time, and doing it only part-time could get you "valley of bad rationality"-type problems.
Third, I suspect you could easily run into problems like what you might get by saying "an ideal reasoner would use Solomonoff Induction, so I should too". That's a wonderful idea, except that even approximating it is computationally insane, and in practice you won't get to use any of the advantages that make Solomonoff Induction theoretically optimal.
If you instead mean things like "an ideal FAI would cooperate in PD-like scenarios given certain conditions", then sure. But again, I'm not sure the FAI heuristic is saving you any work.
The ideal FAI would ignore uncomputable possibilities. Therefore I should too.
A factual FAI might, for mere practical reasons. I don't see why an ideal FAI normatively should ignore them, though.
Ok, thanks.
I also don't see any reason to go from "the FAI doesn't care about identity* to "I shouldn't think identity exists."
I don't either, now that I think about it. What motivated me to make this post is that I realized that I had been making that leap, thanks to applying the heuristic. We both agree the heuristic is bad.
Why are we talking about a bad heuristic? Well, my past self would have benefited from reading this post, so perhaps other people would as well. Also, I wanted to explore the space of applications of this heuristic, to see if I had been unconsciously applying it in other cases without realizing it. Talking with you has helped me with that.
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
= 27d567bc2d48d9f40e9ccc20ea370b15)
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
This is an interesting idea! Some thoughts:
Doesn't acausal trade (like all trade) depend on enforcement mechanisms? I can see how two AI's might engage in counterfactual trade, since they can simulate each other and see that they self-modify to uphold the agreement, but I don't think a human would be able to do it.
Also, I'd like to hear more about motivations for engaging in counterfactual trade. I get the multiverse one, though I think that's a straightforward case of acausal trade rather than a case of counterfactual trade, since you would be trading with a really existing entity in another universe. But can you explain the second motivation more?