It's entirely unclear that we know it when we see it either. Humans don't qualify as a human-friendly intelligence, for example (or us creating an uFAI wouldn't be a danger). We might know something's not it when we see it, but that is not the same thing as knowing something is it when we see it.

The actions of any computable agent - including humans - can be expressed using a utility function.

This is a highly questionable statement concerning humans, and the paper linked from that page doesn't appear to prove it.

Edit: ah, this includes "functions" that anyone else would call a "stupidly complicated state machine" and which may not actually be feasible to calculate.

That too. Freshly-shaved with an electric is like five o'clock shadow with a blade.

YMMV, of course. But I switched to wet shaving from electric when I discovered I could shave without shaking my skull with vibrations.

Many wet shave razors really, really suck.Disposables are really shitty and should be avoided. Try various models. A post on my blog about razors.

Which twenty have the highest number of votes?

Your assessment looks about right to me. I have considerable experience of averagely-incompetent nonprofits, and SIAI looks normal to me. I am strongly tempted to grab that "For Dummies" book and, if it's good, start sending copies to people ...

I think pretty much anything can be turned to stupid purposes if people try hard enough. And they consistently do.

I think the Sequences got everything right and I agree with them completely.

Even Eliezer considers them first drafts (which is, after all, what they were: a two-year exercise in writing raw material for the forthcoming book), not things to be treated as received wisdom.

Yeah, I didn't mean "therefore we should make everyone morbidly obese to raise the general intelligence level", I mean that having obesity indicates you have enough food (since too much food hasn't really been that widespread a problem before this).

As an aside, I seriously think we need to start considering general AI stuff as off-topic again.

+1

It's interesting, but it's not something that fits the tagline: "refining the art of human rationality".