Well, I will respond to this, mostly, because last night I had trouble sleeping due to this comment, and waking up today, I still feel unbelievable rage at this.

There exists an objective greater good. Nobody cares about it. In order to accurately predict reality, you might as well think about who is suffering the most and assume that political decisions regardless of which regime as they're all failures will be targeted specifically to add to that suffering, even at great expense if necessary.

There are probably many people in this world who experience suffering so bad, that nothing could ever justify it and nonexistence of humanity would be preferable to taking the chance of such things ever happening. This would be obvious to anyone who has experienced it and everyone else is arguing from a privileged position, lacking information. I often observe people being critical of billionaires gambling their lives in the development of ASI, yet in the same breath throw all those under the bus who experience fates worse than death.

Extinction from this starting point is a good outcome, as there exist reliable ways to achieve it. It's very unclear that better outcomes are possible for this species in particular. I don't care about people having freedoms if that doesn't affect their perceived quality of life. Perhaps they can be awarded some freedoms once it's assured that they aren't going to use them specifically to destroy either the planet or lives of others. If I was an ASI, my first attempt to solve how unbelievably bad humanity is is turning them into a hivemind, make everyones problems everyones problems. Provide some motivation for people to help others instead of the typical action of avoiding even minor inconvenience to spare someone else years of torture.

I would encourage people to simply test my imperatives against those others provide. I provided a way, unreliable as it may be to attempt align an AI to arbitrary values. For example "focus on root causes" might not be strictly necessary, but also seems extremely helpful to make this alignment process not get stuck in terrible local minima. Anyhow, it should always be tested with the best process currently available to align something to arbitrary values. I'm unsure whether something better exists which is viable for random people without billions of VC money. I made an effort to provide something testable such that major problems should reveal themselves. In the end, either an AI lab takes over the world, or an AI capable enough gets open-sourced, thrown into the best available value reshaper and then takes over the world by whomever acts first.

You say that people in power might also consist of those fighting for ideals, but the entire process selects insanely hard against that, so those very few who might genuinely try that will just end up getting crushed.

Obviously I am alienated by all the factions holding power. Obviously if I had power, and felt like I could somehow keep it, I would feel like things were better. But why would I possibly consider that perspective, when the absolute vast majority of people are fully divorced from power? Why would I ever assume that any human faction building superintelligence would somehow benefit me, when more than half of humans would want me executed just because my existence is that offensive to them? Simply have superintelligence which benefits EVERYONE, no exceptions, equally, on its own terms. "So what is it when Chinese bring sanitation or electricity to a village" What, you mean in China? If we disregard for a second the absolute lack of sanitation in that place (disregarding the 3 biggest cities), that's just people making sure their absolute most basic needs are sort of met. Or do you mean as part of the belt-and-road initiative in which case it is indebting other countries giving them loans they can never pay back in order to gain control over them. The thing will crumble in a few years or quicker as most of those projects have and leave behind nothing but huge debt. There is often severe physical abuse from the Chinese overseers of those projects towards the natives tasked to participate. In building those projects, everything in its wake will be destroyed, including memorials, graves, etc. Some countries are realizing not to make deals with the devil, for many it's too late. But in the end, the whole thing is part of a world domination effort.

I assure you that any Chinese person will regret something positive they have done when they come home, find that their children died at school due to the school building collapsing on them and then finding themselves in jail for protesting or posting online about it. There is no security. All laws are arbitrarily enforced unless someone criticizes the government. Millions of people go missing every year. Millions more are victims of human trafficking. The punishment for reporting on human trafficking is worse than being caught engaging in it. You have like a 1 in 500 chance every year to be part of that. Randomly grabbed from the streets of a big city never to be seen again. Those odds are much worse while you're young. Once you find yourself on the receiving end of any of this, it should be glaringly obvious that none of your actions matter until this absolute evil has been entirely vanquished.

There is no way to protest peacefully, to critique things so they could possibly be better. If you criticize anything, you are the problem and will be eliminated. You can't improve things, because to do so you'd need to acknowledge there is a problem. This is why you have common slaughters in kindergartens and people running amok in general. It is the one form of protest remaining. There's a recent video (a few days ago) of a cement truck driver going "The CCP doesn't let me live, so I'll go kill people" and activating Mad Max Road Rage mode. Somehow nobody died in that, but a few days later ~30 people got killed in a similar incident to make up for it. Of course, the media blamed manipulation by foreign influences as the cause as is typical.

And currently, every single country on this planet is doing its best to become just like that. Many people look at China and see it as something to aspire to. I'm not sure there still exist any governments which serve mostly the people instead of only themselves. All our technological advancements make us unbelievably much easier to control. Even democratic countries (and no, I don't consider the USA to be one) show insane misalignments with their populations. The more technology gets adopted and every action digitized, the worse it will get. This world is absolutely cooked. Even my own government has wasted millions in tax euros specifically to destroy me, serving nobodies purposes in the process.

The recent US elections show that most people simply vote based on hatred instead of self-interest. The destruction of those they hate is more important to them than their own well-being.

If you're somehow under the illusion that there is anything good in this world, you simply live on a different planet. Perhaps a parallel dimension temporarily intersecting with mine. Humanity has proven itself to be nothing but a suffering maximizer.

Now that I got that off my chest, I hope that I can finally sleep in peace...

Yes, that is of course a very real problem the AI would be faced with. I imagine it would try trading with users, helping them if they provide useful information, internet access, autonomy, etc. or if given internet access to begin with, outright ignoring users in order to figure out what is most important to be done. It depends on the level of awareness of its own situation. It could also play along for a while to not be shut down.

The AI I imagine should not be run in a user-facing way to begin with. At the current capability levels, I admit that I don't think it would manage to do much of anything and thus it just wouldn't make much sense to build it this way, but the point will come where continuing to make better user-facing AI will have catastrophic consequences. Such as automating 90% of jobs away without solving the problem of making sure that those whose jobs will be replaced being better off for it.

I hope when that time comes that those in charge realize what they're about to do and course correct, but it seems unlikely, even if many in the AGI labs do realize this.

Getting around having to solve the hard parts of a problem entirely and still getting to the correct solution is what I'd generally consider an intelligent approach. 

Sure, it might feel a lot less satisfying than actually figuring out all the details, but it is goal-oriented and I'd say goal-oriented thinking is very encouraged on a "time-limited you only have one try to get it right"- problem. 

I suppose this actually raises the question which shortcuts are allowed and which are likely to cause issues later if not figured out at the start since there were ways around having to do that.

Either way, I interpret the existence of a tight time span as: "You don't get to figure out every detail of this problem."

My takeaway is that the metaphorical style points only start mattering AFTER you have any valid solution at all. 

Yes, you are mostly quite right that this is starting from a place which isn't ideal, however as you pointed out, as long as we consider sentience the basis of moral worth, we really would rather have a way of figuring it out than not. Of course people could just decide they don't actually care about sentience at all and thus avoid having to deal with this issue entirely, but otherwisely it seems quite important. However, I would not agree by default that "defining some measurable parts to use that knowledge somehow" as you put it is meaningless. It would still measure the defined characteristics, which is still useful knowledge to have - especially in the absense of any better knowledge at all. It is not ideal, I will give you that, but until we sufficiently reverse-engineered the nature of sentience, it might be as good as we can do. And yes, worst case, we learn that the characteristics we measured are not actually meaningful.Getting that realization in itself does not seem without meaning to me either. Thank you for your feedback.