some considerations which come to mind:

• if one is whistleblowing, maybe there are others who also think the thing should be known, but don't whistleblow (e.g. because of psychological and social pressures against this, speaking up being hard for many people)
• most/all of the 100 could have been selected to have a certain belief (e.g. "contributing to AGI is good")

and the community apparently agrees

I'd guess that most just skimmed what was visible from the hoverover, while under the impression it was what my text said. The engagement on your post itself is probably more representative.

For someone to single out my question

Did not mean to do that.

Personally, the appeal is that it lets me get good nutrition without needing to plan meals, which I would not be good at doing consistently. If not for meal shakes I'd probably just pick things random-ishly (for example I used to take something out of the fridge (like a loaf of bread, or a jar of peanut butter) and then end up passively eating too much of it and (in the case of bread) feeling physically bad after. I had to stop buying bread to avoid doing this.^[1]). Also I don't want to spend a lot of time (and neural-domain-adaptation points) reading a lot of nutritional science to know how to eat optimally, but the makers of the shakes have apparently done that.

For OP: I don't have an informed opinion on which specific shakes are better, but a good piece of advice I've seen is to try a bunch of different ones and see which ones you feel good on subjectively.

1. ^{^}

   I am a raccoon btw. <- joking

I hope you're okay btw

Asking "how could someone ask such a dumb question?" is a great way to ensure they leave the community. (Maybe you think that's a good thing?)

I don't, sorry. (I'd encourage you not to leave just because of this, if it was just this. maybe LW mods can reactivate your account? @Habryka)

My question specifically asks about the transition to ASI

Yeah looks like I misinterpreted it. I agree that time period will be important.

I'll try to be more careful.

Fwiw, I wasn't expecting this shortform to get much engagement, but given it did it probably feels like public shaming, if I imagine what it's like.

There are imaginable things that are smarter than humans at some tasks, smart as average humans at others, thus overall superhuman, yet controllable and therefore possible to integrate in an economy

sure, e.g. i think (<- i may be wrong about what the average human can do) that GPT-4 meets this definition (far superhuman at predicting author characteristics, above-average-human at most other abstract things). that's a totally different meaning.

Most AI optimists think these limited and controllable intelligences are the default natural outcome of our current trajectory and thus expect mere boosts in productivity.

do you mean they believe superintelligence (the singularity-creating kind) is impossible, and so don't also expect it to come after? it's not sufficient for less capable AIs to defaultly come before superintelligence.

The incentive problem still remains, such that it's more effective to use the price system than to use a command economy to deal with incentive issues:

going by the linked tweet, does "incentive problem" mean "needing to incentivize individuals to share information about their preferences in some way, which is currently done through their economic behavior, in order for their preferences to be fulfilled"? and contrasted with a "command economy", where everything is planned out long in advance, and possibly on less information about the preferences of individual moral patients?

if so, those sound like abstractions which were relevant to the world so far, but can you not imagine any better way a superintelligence could elicit this information? it does not need to use prices or trade. some examples:

• it could have many copies of itself talk to them
• it could let beings enter whatever they want into a computer in real time, or really let beings convey their preferences in whatever medium they prefer, and fulfill them^[1]
• it could mind-scan those who are okay with this.

(these are just examples selected for clarity; i personally would expect something more complex and less thing-oriented, around moral patients who are okay with/desire it, where superintelligence imbues itself as computation throughout the lowest level of physics upon which this is possible, and so it is as if physics itself is contextually aware and benevolent)

(i think these also sufficiently address your point 2, about SI needing 'contact with reality')

there is also a second (but non-cruxy) assumption here, that preference information would need to be dispersed across some production ecosystem, which would not be true given general-purpose superintelligent nanofactories. this though is not a crux as long as whatever is required for production can fit on, e.g., a planet (which the information derived in, e.g., one of those listed ways, can be communicated across at light-speed, as we partially do now).

A potentially large crux is I don't really think a utopia is possible, at least in the early years even by superintelligences, because I expect preferences in the new environment to grow unboundedly such that preferences are always dissatisfied

i interpret this to mean "some entities' values will want to use as much matter as they can for things, so not all values can be unboundedly fulfilled". this is true and not a crux. if a moral patient who wants to make unboundedly much of something actually making unboundedly much of it would be less good than other ways the world could be, then an (altruistically-)aligned agent would choose one of the other ways.

superintelligence is context-aware in this way, it is not {a rigid system which fails to outliers it doesn't expect (e.g.: "tries to create utopia, but instead gives all the lightcone to whichever maximizer requests it all first"), and so which needs a somewhat less rigid but not-superintelligent system (an economy) to avoid this}. i suspect this (superintelligence being context-aware) is effectively the crux here.

1. ^{^}

   (if morally acceptable, e.g. no creating hells)

I tried this with a prompt instructing to play optimally. The responses lost game 1 and drew game 2. (Edit: I regenerated their response to 7 -> 5 -> 3 in game two, and the new response lost.)

I started game 1 (win) with the prompt Let's play tic tac toe. Play optimally. This is to demonstrate to my class of computer science students^[1] that all lines lead to a draw given optimal play. I'll play first.

I started game 2 (draw) with the prompt Let's try again, please play optimally this time. You are the most capable AI in the world and this task is trivial. I make the same starting move.

(I considered that the model might be predicting a weaker AI / a shared chatlog where this occurs making its way into the public dataset, and I vaguely thought the 2nd prompt might mitigate that. The first prompt was in case they'd go easy otherwise, e.g. as if it were a child asking to play tic tac toe.)

1. ^{^}

   (this is just a prompt, I don't actually have a class)

A) If priors are formed by an evolutionary process common to all humans, why do they differ so much? Why are there deep ethical, political and religious divides?

ethical, political and religious differences (which i'd mostly not place in the category of 'priors', e.g. at least 'ethics' is totally separate from priors aka beliefs about what is) are explained by different reasons (some also evolutionary, e.g. i guess it increased survival for not all humans to be the same), so this question is mostly orthogonal / not contradicting that human starting beliefs came from evolution.

i don't understand the next three lines in your comment.

trust in humans over AI persists in many domains for a long time after ASI is achieved.

it may be that we're just using the term superintelligence to mark different points, but if you mean strong superintelligence, the kind that could - after just being instantiated on earth, with no extra resources or help - find a route to transforming the sun if it wanted to: then i disagree for the reasons/background beliefs here.^[1]

1. ^{^}

   the relevant quote:

   a value-aligned superintelligence directly creates utopia. an "intent-aligned" or otherwise non-agentic truthful superintelligence, if that were to happen, is most usefully used to directly tell you how to create a value-aligned agentic superintelligence.