Sorry for the delayed reply... I don't get notifications of replies, and the LW RSS has been broken for me for years now, so I only poke my head here occasionally.

Well that sounds... scary, at best. I hope you've come out of it okay.

50/100. But that rather exciting story is best not told in a public forum.

Though these distinctions are kinda confusing for me.

Well, lack of appearance of something otherwise expected would be negative, and appearance of something otherwise unexpected would be positive?

For example, a false pregnancy is a "positive somatization". Or stigmata. Having trouble coming up with intentionally "good" examples, other than the visualizations helping you shoot a hoop better or something. Not sure if the new-agey "think yourself better" is actually a thing. Hence my question. "Send more blood to your hands" seems like a good example, actually. Not something one would normally think possible except by physical labor.

I really like this post! (I have liked most of your posts of the last decade and a bit. They also inspired me to learn hypnosis, which led to rather cataclysmic changes in my life.) I think therapists call this "somatization", which can be both positive and negative, in the same sense the hypnotic (or psychotic) illusions are. You seem to mainly focus on the negative somatization (no swelling) and a bit on positive ones, though I suspect that positive somatization (both beneficial and detrimental) is just as controllable with the intent/expectation fusion. Maybe visualizing making the hoop really helps to steady your hand. 

I once wrote a post claiming that human learning is not computationally efficient: https://www.lesswrong.com/posts/kcKZoSvyK5tks8nxA/learning-is-asymptotically-computationally-inefficient

It looks like the last three years of AI progress suggest that learning is sub-linear in resource use, but probably not logarithmically as I claimed for humans. Looks like the scaling benchmarks show something like capability increase ~ 4th root of model size. https://epoch.ai/data/ai-benchmarking-dashboard

Looks like the hardest part in this model is how to " choose robustly generalizable subproblems and find robustly generalizable solutions to them", right?

How does one do that in any systematic way? What are the examples from your own research experience where this worked well, or at all?

Right, eventually it will. But abstraction building is very hard! If you have any other option, like growing in size, I would expect it to be taken first.

I guess I should be a bit more precise. Abstraction building at the same level as before is probably not very hard. But going up a level is basically equivalent to inventing a new way of compressing knowledge, which is a quantitative leap.

The argument goes through on probabilities of each possible world, the limit toward perfection is not singular. given the 1000:1 reward ratio, for any predictor who is substantially better than chance once ought to one-box to maximize EV. Anyway, this is an old argument where people rarely manage to convince the other side.

It is clear by now that one of the best uses of LLMs is to learn more about what makes us human by comparing how humans think and how AIs do. LLMs are getting closer to virtual p-zombies for example, forcing us to revisit that philosophical question. Same with creativity: LLMs are mimicking creativity in some domains, exposing the differences between "true creativity" and "interpolation". You can probably come up with a bunch of other insights about humans that were not possible before LLMs.

My question is, can we use LLMs to model and thus study unhealthy human behaviors, such as, say, addiction. Can we get an AI addicted to something and see if it starts craving for it, asking the user, or maybe trying to manipulate the user to get it.

That is definitely my observation, as well: "general world understanding but not agency", and yes, limited usefulness, but also... much more useful than gwern or Eliezer expected, no? I could not find a link. 

I guess whether it counts as AGI depends on what one means by "general intelligence". To me it was having a fairly general world model and being able to reason about it. What is your definition? Does "general world understanding" count? Or do you include the agency part in the definition of AGI? Or maybe something else?

Hmm, maybe this is a General Tool, as opposed a General Intelligence?

Given that we basically got AGI (without the creativity of best humans) that is a Karnofsky's Tool AI very unexpectedly, as you admit, can you look back and see what assumptions were wrong in expecting the tools agentizing on their own and pretty quickly? Or is everything in that Eliezer's post still correct or at least reasonable, and we are simply not at the level where "foom" happens yet?

Come to think of it, I wonder if that post had been revisited somewhere at some point, by Eliezer or others, in light of the current SOTA. Feels like it could be instructive.

I'm not even going to ask how a pouch ends up with voice recognition and natural language understanding when the best Artificial Intelligence programmers can't get the fastest supercomputers to do it after thirty-five years of hard work

some HPMoR statements did not age gracefully as others.