I'm convinced of utilitarianism as the proper moral construct, but I don't think an AI should use a free-ranging utilitarianism, because it's just too dangerous. A relatively small calculation error, or a somewhat eccentric view of the future can lead to very bad outcomes indeed.

A really smart, powerful AI, it seems to me, should be constrained by rules of behavior (no wiping out humanity/no turning every channel into 24-7 porn/no putting everyone to work in the paperclip factory), The assumption that something very smart would necessarily reach correct utiltarian views seems facially false; it could assume that humans must think like it does, or assume that dogs generate more utility with less effort due to their easier ability to be happy, or decide that humans need more superintelligent machines in a great big hurry and should build them regardless of anything else.

And maybe it'd be right here or there. But maybe not. I think almost definitionally that FAI cannot be full-on, free-range utilitarian of any stripe. Am I wrong?

I believe you can strip the AI of any preferences towards human utility functions with a simple hack.

Every decision of the AI will have two effects on expected human utility: it will change it, and it will change the human utility functions.

Have the AI make its decisions only based on the effect on the current expected human utility, not on the changes to the function. Add a term granting a large disutility for deaths, and this should do the trick.

Note the importance of the "current" expected utility in this setup; an AI will decide whether to industrialise a primitive tribe based on their current utility; if it does industrialise them, it will base its subsequent decisions on their new, industrialised utility.

me too

There is a common intuition and feeling that our most fundamental goals may be uncertain in some sense.

In what follows, I will naturalistically explore the intuition of supergoal uncertainty.

These are entirely too representative of this post. I admit it's possible I lack adequate background, but this post seems incredibly dense and convoluted. I literally do not know what you're talking about, and I have enough external evidence of my reading comprehension to conclude that it's significantly the author's fault. The idea may be clear in your mind, but you need to spell it out in clear and simple terms if you want others to follow you. Defining "supergoal uncertainty" would be a necessary step, though it would still be well short of sufficient.

this post seems incredibly dense and convoluted. I literally do not know what you're talking about

That was not my experience. I understood everything in the first five paragraphs without having to reflect or even read a second time except that I did have to reflect for a few minutes on the last sentence of paragraph four. Although I am still less confident that I know what Justin intended there than I am with the other sentences, I am 72% confident I know. I think he meant that even if we are not religious, society tends to pull us into moral realism even though of course moral realism is an illusion. (Time constraints prevent me from reading the rest now.)

Defining "supergoal uncertainty" would be a necessary step

Oh, he did that. And the definition was quite clear to me on first reading, but then I have done a lot of math, and a lot of math in which I attempt my own definitions.

There also appear to be outright misuses of vocabulary, unless there are technical meanings I am unaware of. I.e. "I may soon post and explore the effects of supergoal uncertainty in its various reifications on making decisions."

Not even the most obscure continental philosophy gets away with using 'reify' that way.

Still, it looks like there might be some interesting ideas somewhere in there.

Some things I use to test mental ability as well as train it are: BrainWorkshop (A free dualNback program), Cognitivefun.net (A site with assorted tests and profiles including everything from reaction time, to subitizing, to visual backward digit span), Posit Science's jewel diver demo (a multi-object tracking test), and Lumosity.com (brainshift, memory matrix, speed match, top chimp. All of these tests can be found for free on the internet).

Subjectively the regular use of these tests has increased my metacognitive and self monitoring ability. Anyone have other suggestions? How about tests one can do without the aid of external devices?

In complement to determining whether one's brain isn't in its best state there is the question of how to improve or fix it. Keeping with the general spirit of this thread, what are some strategies people use to improve their cognitive functioning (as it pertains to low level properties such as short term memory) in the short term without the use of external aids? A few I use are priming emotional state with posture, expression, and words, doing mental arithmetic, memorizing arbitrary information, and doing the above mental tests.

Surely, to label a statement "vague" is a higher order of insult than to call it "wrong". Newton was wrong but at least he was not vague.

Statements should be as precise as possible, but no more precise.

Many highly intelligent people are poor thinkers. Many people of average intelligence are skilled thinkers. The power of a car is separate from the way the car is driven.

-- Edward de Bono