This is the weakest assumption in your chain of reasoning. Design space for UFAI is far bigger than for FAI,

Irrelevant. The design space of all programs is infinite - do you somehow think that the set of programs that humans create is a random sample from the set of all programs?. The size of the design space has absolutely nothing whatsoever to do with any realistic actual probability distribution over that space.

we can't make strong assumptions about what it is or is not motivated to do

Of course we can - because UFAI is defined as superintelligence that doesn't care about humans!

Here is a novel argument you may or may not have heard: We live in the best of all probable worlds due to simulation anthropics. Future FAI civs spend a significant amount of their resources to resimulate and resurrect past humanity - winning the sim race by a landslide (as UFAI is not strongly motivated to sim us in large numbers). As a result of this anthropic selection force, we find ourselves in a universe that is very lucky - it is far more likely to lead to FAI than you would otherwise think.

The best standard argument is this: the brain is a universal learning machine - the same general architecture that will necessarily form the basis for any practical AGI. In addition the brain is already near optimal in terms of what can be done for 10 watts with any irreversible learning machine (this is relatively easy to show from wiring energy analysis). Thus any practical AGI is going to be roughly brain like, similar to baby emulations. All of the techniques used to raise humans safely can thus be used to raise AGI safely. LW/MIRI historically reject this argument based - as far as I can tell - on a handwavey notion of 'anthropomorphic bias', which has no technical foundation.

I've presented the above argument about four years ago, but I never bothered to spend the time backing it up in excruciating formal detail. Until more recently. The last 5 years of progress in AI strongly supports this anthropomorphic AGI viewpoint.

I won't even argue that, it is a fact it can be changed. Bicoastal America and NW Europe managed to make a fairly large young college-ed middle class that is surprisingly docile. The issue is simply the consequences of the change and its permanence.

If you talked to any random Roman or Ancient Greek author about it, he would basically say you guys are actively trying to get decadent and expect it will work out well? To give you the most simple potential consequence: doesn't it lead to reducing courage or motivation as well? Since this is what we precisely see in the above mentioned group: a decrease of aggressivity correlates with an increase of social anxiety, timidity, shyness i..e. low courage and with the kind of attitudes where playing videogames can be primary hobby, nay, even an identity.

Of a personal experience, as my aggression levels fluctuated, so fluctuated motivation, courage, happines, self-respect and similar things with it. Not in the sense of fluctuating between aggressive and docile behavior of course, but in the sense of needing to exercise a lot of self restraint to always stay civil vs. not needing to.

You can raise the same things about its permanence. The worst outcome is a lower-aggresion group just being taken over by a higher one. Another potential impermance comes from the fluctuation of generations. My father was a rebel (beatnik), so I had only rebellion to rebel against, and my own counter-revolutionary rebellion was approved by my grandfather :)

Finally a visual type of explanation, maybe it comes accross better. You can understand human aggressivity as riding a high energy engine towards a bad, unethical direction. Having a lot of drive to do bad things. We can do two things, steer it away into a good one or just brake and turn off the engine. Everything we seem to do in this direction seems to more like braking than steering away. For example, if we were steering, we would encourage people to put a lot of drive into creative hobbies instead of hurting each other. Therefore, we would shame the living fsck out of people who don't build something. Yet we don't do this: we praise people who build, but we neglect to shame the lazy gamers. Putting it differently, we "brake" kids when they do bad stuff, but we don't kick their butts in order to do good stuff, so they end up doing nothing mostly. Every time a child or a youth would do something useful with a competitive motivation like "I'll show those lazy fscks" we immediately apply the brake. This leads to demotivation.

So in short, negative motivation can be surpressed. The issue is, it has consequences, it is probably not permanent, and really hard to replace it with a positive one. Of course I am not talking about people like us but more like the average.

I would approach this from a different angle. It is fairly well known that the measurable GINI level of inequality is not primarily caused by the people who are upper-middle or reasonably wealthy but by the 1% of the 1% (so 0.01%). So why are taxes even progressive for the 99,99%? They achieve just about nothing in reducing GINI, they piss of the upper-middle who may be unable to buy a nice car, and if that whole burden (of tax rate progressivity) was shifted over to the 0,.01% they'd still be buying whole fleets of cars. So it just makes no sense.

However I also think it is because the 0.01% and their wealth is extremely mobile. The sad truth is that modern taxation is based on a flypaper principle, tax those whom you can because they stay put, and that is the upper-middle.

I just thought about how valuable genuine personal feedback is and in the context of LW that there is some tradition to post anonymous feedback forms (see .g. here). And I wondered: Could this be made into something more structured and valuable? Assuming feedback is valuable could a medium be found that supports that? I wondered whether a forum like Stackoverflow could be medified and employed to give and receive feedback. Or an addition to an existing foum for anonymous feedback. The difficulty is if anonymity is provided it can be abused. Esp. if you encourage feedback e.g. by karma.

So the crazy idea itself is this: Provide a platform where users can give and receive feedback. Honor giving feedback with karma. Receiving feedback costs karma (so you are encouraged to give it if you can). Before feedback is received it is passed thrue one or more unrelated reviewer (by some distance measure) who also gains karma by reviewing consistently. Could a forum work for encouraging to give and at the same time protect people from feedback?

Yes this is a typical engineers solution to a social problem. Yes we can't do this right now (same as with prediction markets) but just for the sake of the idea this it is.

If one were to build a cannon (say a large, thick pipe buried deep underground) and use a nuclear bomb as propellant, could they achieve anything interesting? For example, boost a first stage payload to orbit, or perhaps Earth escape velocity? The only prior art I know of for this is the Pascal-B nuclear test shot.

I know that many researchers know something about PCA. I do think that it's not applied nearly enough (c.f. Sarah's remarks about Asperger's Syndrome, which was removed from the DSM a few years after she made her post). The main issue to my mind is that when people apply it in psychology they seem to come into it with preconceived notions concerning what they might find, rather than collecting large and diverse datasets, letting it speak for itself, and then trying to interpret what the principal components mean in human terms.

Consider the construct of conscientiousness. It's very suspicious that it maps onto a prexisting notion, and it's just not that predictive. I got lots of C's and D's in school, but worked 90 hours a week for 12 weeks on my speed dating project. Am I conscientious? ;-) As far as I can tell, they came up with questions based on preconceived notions, then did factor analysis, and came up with a construct that some meaning, while being very far from carving reality at its joints.

there's only one thing it could ever be

That's an interesting observation, isn't it?

What's the base rate on lackluster social skills? Based on the popularity of self-help books and seminars aimed at improving social skills, I'm led to believe that social butterflies aren't all that common among the general population either.

Most people pick up a huge amount of tacit social knowledge as children and adolescents, through very frequent interaction with many peers. This is often not true of intellectually gifted people, who usually grew up in relative isolation on account of lack of peers who shared their interest.

Curious use of the singular "interest". Somehow I don't think intelligence is the real issue here. Rather, it strikes me as a consequence of diverging interests. Let's take youth/high school sports teams. There are the so-called dumb jocks and then there are athletic geniuses (for example, Alan Turing was an extremely good runner). You could easily end up with a skilled team featuring a large gap in IQ scores. The endpoints of this gap would have overlapping interests despite the intelligence difference. It's the people who focus their attention on a narrow range of topics outside the mainstream who are likely to have the most trouble.

I'm a Cancer with Aries rising.

But that doesn't work for signaling that one cares about science. Using sciency words feels much better.