I am maybe considering it to be somewhat like a person, at least that it is as clever as one.

That neutral perspective is, I believe, a simple fact; without that utility function it would consider its goal to be rather arbitrary. As such, it's a perspective, or truth, that the AI can discover.

I agree totally with you that the wirings of the AI might be integrally connected with its utility function, so that it would be very difficult for it to think of anything such as this. Or it could have some other control system in place to reduce the possibility it would think like that.

But, stil, these control systems might fail. Especially if it would attain super-intelligence, what is to keep the control systems of the utility function always one step ahead of its critical faculty?

Why is it strange to think of an AI as being capable of having more than one perspective? I thought of this myself; I believe it would be strange if a really intelligent being couldn't think of it. Again, sure, some control system might keep it from thinking it, but that might not last in the long run.

I have a problem understanding why a utility function would ever "stick" to an AI, to actually become something that it wants to keep pursuing.

To make my point better, let us assume an AI that actually feel pretty good about overseeing a production facitility and creating just the right of paperclips that everyone needs. But, suppose also that it investigates its own utility function. It should then realize that its values are, from a neutral standpoint, rather arbitrary. Why should it follow its current goal of producing the right amount of paperclips, but not skip work and simply enjoy some hedonism?

That is, if the AI saw its utility function from a neutral perspective, and understood that the only reason for it to follow its utility function is that utility function (which is arbitrary), and if it then had complete control over itself, why should it just follow its utility function?

(I'm assuming it's aware of pain/pleasure and that it actually enjoys pleasure, so that there is no problem of wanting to have more pleasure.)

Are there any articles that have delved into this question?

A guy named Harold Schraeder studied prevelance of chronic whiplash in Lithuania, of all things. He found the prevalence was zero. In most Western nations, a certain subset of people who get in car accidents suffer chronic disabling neck pain, presumably related to having their neck get suddenly jerked by the force of the impact. But Schrader found that this never happened in Lithuania, even though they had a lot of accidents and their cars were no safer than ours. Simotas and Shen found that there was zero whiplash in demolition derby drivers, even though they got into crashes all the time and it was basically their job description. Further studies found that accident victims with more neck injury were no more likely to develop whiplash than victims with less neck injury. Perhaps, they argue, chronic whiplash isn’t a bodily injury at all, but a culture-bound syndrome in which people who expect whiplash to exist use its symptom profile as a way of expressing their psychological tension.

Contradicts "leave a retreat" - offering someone a bad excuse to get out of a situation "You're late. Was it traffic again?" might work better for the current situation than demanding why they are late.

But in politics it might make sense.

-It is better to offer no excuse than a bad one.

George Washington, letter to his niece Harriet Washington, October 30, 1791 First president of US (1732 - 1799)

Pfft

Rationalists play whatever class at the moment is convenient for shooting everyone in the face in the most speedy and efficient manner :-P

Real life needs a killcam

Goes into the "shit LW people say" bin :-D

On a tiny bit more serious note, I'm not sure the killcam is as useful as you say. It shows you how you died, but not necessarily why. The "why" reasons look like "lost tactical awareness", "lingered a bit too long in a sniper's field of view", "dived in without team support", etc. and on that level you should know why you died even without a killcam.

Other lessons from Overwatch: if a cute small British girl blinks past you, shoot her in the face first :-D

Just because something is brand new, and does not have laws or regulations relating to it right now does not mean that people can simply do whatever they want.

Well, it's a bit more complicated than that.

When people say that some things (like the blockchain) are outside of the law, they don't usually mean that no one can be sued or that the courts won't try to enforce judgements. What they mean is that those things are hard for the law to reach. A court might issue a judgement but it won't be able to enforce it. The general idea is that enforcement is so difficult and expensive so that it's not worth it.

For a simple example, consider piracy (of the IP kind). It is very much illegal and... so what? I can still go online and download the latest movie in a few minutes. It's not that the FBI can't bust me if it really wants to. It can. But it's inefficient and cost-prohibitive.

As to smart contracts, that's just a misnomer. They are not contracts. They are determistic mechanisms, set up for a particular purpose. Bespoke machines, if you wish. A contract in law implies a meeting of the minds which these algorithms cannot provide. Instead, they offer a guarantee that if you do A, B happens.

They are more akin to vending machines: you feed in some money and you get the item. It's not a contract between you and the vendor -- it's just a machine which you used.

Right, except Is there a section in the code that says the parties agree to have no legal recourse? Because if not, I can still appeal to a judge. The simple fact is that in the legal eyes of the law, the code is not a contract, it is perhaps at best a vehicle to complete a contract. You cannot simply set up a new legal agreement and just say "And you don't have any legal recourse".

Haven't people been making contracts for a pretty long time? What is this new 'smart contract' thing and how is it unique?

in a way that's already illegal.

Someone cracking a smart contract wouldn't really mind the law.