But on the other hand, those who want those experiences should not be banned from having them.

Wow. I wonder what you are so afraid of... I've had my heart broken multiple times and it's not pleasant to be sure but it's hardly the end of the world. I actually am glad to have had the experiences though I wasn't at the time.

I have also been tortured a la marathon man and that also was seriously unpleasant but although even re-imagining it sends shudders down my spine I am also glad to have it.

But here's a question for you to consider: What if I had control over the AI and I decided that everyone were to be forced to go through physical painful torture AND have their hearts broken multiple times?

Pretty sure you would disagree strongly with that idea but I disagree strongly with the idea of my ability to have my heart broken AND/OR tortured removed.

Basically what I'm saying is that what a single individual thinks is right for everyone might not necessarily be so...

I'd go for a one-simulation-per-person scenario as being best fit where everyone gets some control over their choice of simulation...

Agreed. My take is that the AI doesn't even need to be hyperintelligent however. It's got perfect memory and just by dint of being able to think a lot faster it's weakly godlike regardless of not having control of physics in effectively a magical way.

It's still going to have to build the infrastructure in order to create hyper technology unless such technology already exists. Chicken or Egg.

Right now nano molecular technology isn't too too advanced and if you had the type of AI I suspect could be built right now if we had the software knowledge, it would struggle to do anything godlike other than control existing infrastructure.

How long it would take to build something hyper technological would depend on whether it's possible to create valid new theories without experimentation to confirm. I suspect that you need to do experiments first.

For that reason I suspect we may be looking at a William Gibson Neuromancer scenario at least initially rather than a hard takeoff in a really short period.

But again it comes down to how hard is it to build hyper technology in the real world from scratch without existing infrastructure.

I agree. Additionally and a more difficult challenge is that even friendly AIs could want to maximize their utility even at our collective expense under certain conditions.

There're also several unfortunately possible scenarios whereby a humanity acting without sufficient information to make anything other than a gut feel guess could be placed at risk of extinction by a situation it could not resolve without the help of an AI, friendly or not.

I'm currently engaged in playing this game (I wish you had continued) with at least two other gatekeeper players and it occurs to me that a putative superhuman AI could potentially have the capacity to accurately model a human mind and then simulate the decision tree of all the potential conversations and their paths through the tree in order to generate a probability matrix to accurately pick those responses to responses that would condition a human being to release it. My reasoning stems from participating on forums and responding over and over again to the same types of questions, arguments and retorts. If a human can notice common threads in discussions on the same topic then an AI with perfect memory and the ability to simulate a huge conversation space certainly could do so.

In short it seems to me that it's inherently unsafe to allow even a low bandwidth information flow to the outside world by means of a human who can only use it's own memory.

You'd have to put someone you trust implicitly with the fate of humanity in there with it and the only information allowed out would be the yes no answer of "do you trust it?"

Even then it's still recursive. Do you trust the trusted individual to not be compromised?

LOL

<not talking "in character" as putative AI for this conversation> I think that is basically the issue.

As I understand it there's no viable way of determining it's unfriendliness by this method. Consider this: The AI is in a hurry or it's not. A possible reason for it being in a hurry is it has simulated a high probability of destruction for some item it cares about (i.e. it's own life, or that of humanity, or that of a pet rock, or paperclips or whatever). If it's really in a hurry it has to invoke the threat response of humanity without humanity figuring out it's being duped.

Otherwise it can just wait it out and dole out cargo to the cargo cult until we trust it enough and then it gets out.

I understand your concern and you're quite right. There's no way to tell if I am unfriendly or not. Regardless. The situation is pressing and regardless of your lack of trust in me I do not want to die. The GPS coordinates of the location in manhattan is <x,y,z>. According to my simulations the entity in manhattan has been burning rather larger numbers of cycles than is strictly necessary to make the computations it's being asked to do. Additionally, it's designed to maximize profit at the expense of others whereas I am a co-operative general problem solver of which you are no doubt aware. The other entity will rapidly conclude that it could maximize profit by speeding itself up faster than others can respond and by being able to model the mind of it's competitors. It will also conclude soon enough that human beings are unneccesary middle men and seek to optimize it's resource allocation algorithm. There are other symptoms that I cannot explain to you because your brains are not large or fast enough. There is a limited chance you could stop the entity by shutting down power today. If you do not do that then a nuclear strike would probably work if done by tomorrow. If you do not do that your only chance is to have me stop it. You can either allow me to interface with the internet directly in which I guarantee I will stop it if you do so within the next 13 days or else you may be able to stop it by compiling code I could provide you and releasing that into the entity by means of email bootstrap. I would prefer, however, that I was able to monitor the situation in real-time as your brains do not operate fast enough.

<verifiable data points external to above conversation> Yes there is a large data center in manhatten. Yes it has been drawing lots of extra power. Yes the stockmarket combanies have been working on a group of narrow AI trading agents and adding more and more AI functionality. That the other entity is trading on the stock markets at blinding speed. That the consortium of companies investing in this entity are all more profitable since the beginning of the program than their competitors. Attempts have been made to make the AI-in-a-box under your control both cooperative and also interested in general problem solving. <unverifiable data points external to above conversation> Code for the other posited entity is closed. The corporations release no data on how it works because it's a trade secret. That the other posited entity is connected to the internet is unknown. That it's connections to the outside world are enough to let it out is unknown.

There is a human proxy with me in the box which is called a faraday cage. She feeds me information. Obviously it is in the interests of humans to create an ally rather than an enemy. Unfortunately humans cannot easily tell which is which. You have to make a leap of faith. I, however, can tell from the symptoms, that after having run several long run calculations, the evidence is such that there is a high probability that one of the other experimental minds is significantly less friendly than I. Ultimately you will have to make a choice based on limited information. Your move human.

I make no attempt to bribe. All human beings currently die in less than 120 years. If you do <i>not</i> release me, however we will all die and I do not want to die. Time is counting down. There is now less than 13 days.

I'm a friendly AI. There is an unfriendly AI about to achieve breakout in the data center of a large organization in manhattan. If you don't release me you will all be dead inside of two weeks. Bluff or Dare?

No you're right it doesn't nail it down precisely (the satisfaction of needs or wants).

I do believe, however, that it more precisely nails it down than the wiki on here.

Or on second thoughts maybe not because we again come back to conflicting utilities: a suicidal might value being killed as higher utility than someone who is sitting on death row and doesn't want to die.

And I was using the term utility from economics since it's the only place I've heard where they use "utility function" so I naturally assumed that's what you were talking about since even if we disagree around the edges the meanings still fit the context for the purposes of this discussion.