I agree with the problem (which is rumoured to be one factor in the Trump regime's unhinged behaviour), but I doubt that an add-on to improve LLM output will do any more than polish the turd. And how would such an add-on be created? More LLMs would only pile the shit higher. Even humans don't seem capable of "revising" LLM slop into anything useful.

ETA: "anything useful" depends on one's use for it. Clearly (if the rumours are true) it's being very useful to the Trump regime. They can do anything they like and trot out an LLM argument for it, and do the same to answer all possible reactions and all subsequent events. What people seem incapable of doing is "revising" LLM output into something possessing epistemic virtue.

It’s like asking why high kinetic energy “feels” hot. It doesn’t, heat is just how the brain models signals from temperature receptors and maps them into the self-model.

We know how high (random) kinetic energy causes a high reading on a thermometer.

We do not know why this "feels hot" to people but (we presume) not to a thermometer. Or if you think, as some have claimed to, that it might actually "feel hot" to a strand of mercury in a glass tube, how would you go about finding out, given that in the case of a thermometer, we already know all the relevant physical facts about why the line lengthens and shrinks?

Sections 4 and 5 explain why this evolved: it’s a useful way for the brain to prioritize action when reflexes aren’t enough. You “feel” something because that’s how your brain tracks itself and the environment.

This is redefining the word "feel", not accounting for the thing that "feel" ordinarily points to.

The same thing happened to the word "sensation" when mechanisms of the sensory organs were being traced out. The mechanism of how sensations "feel" (the previous meaning of the word "sensation") was never found, and "sensation" came to be used to mean only those physical mechanisms. This is why the word "quale" (pl. qualia) was revived, to refer to what there was no longer a name for, the subjective experience of "sensations" (in the new sense).

The OP, for all its length, appears to be redefining the word "conscious" to mean "of a system, that it contains a model of itself". It goes into great detail and length on phenomena of self-modelling and speculations of why they may have arisen, and adds the bald assertion, passim, that this is what consciousness is. The original concept that it aims and claims to explain is not touched on.

Where does this model fail?

I didn't see any explanation of why subjective experience exists at all. Why does it feel like something to be me? Why does anything feel like something?

No, I read your vignette as describing a process of things snowballing all on their own, rather than by any such skilful response on either side. Hence my sceptical reply to it.

This is a very strange read, for two reasons.

The story began (emphasis added) (ETA: more emphasis added):

When the cashier smiles at you 1% more than usual, you probably don't stop and wonder whether it's a sign or not. You won't think anything of it because it's well within the noise -- but you might smile 1% more in return without noticing that you do.

And I took that to be the pattern of the subsequent mutual 1%-ing, neither of the participants noticing what they are doing until you envisage some outside witness waking them up:

Before you know it people might be saying "Get a room, you two!".

Of course there are skills. But they all begin with noticing.

I am claiming no particular social skills for myself, only perhaps a general skill of noticing.

"Frog boiling" is standing in for "responding skillfully to women expressing subtle interest, and managing to turn it into clear cut interest so that asking her out is no longer a leap of faith"... right?

No, I read your vignette as describing a process of things snowballing all on their own, rather than by any such skilful response on either side. Hence my sceptical reply to it.

Am I reading this correctly that you're patting yourself on the back

No.

Is accidentally intentionally getting women too obviously interested in them the problem that you think most men have in dating?

No, that strikes me as so far fetched a scenario as to only occur in the fiction of another era.

Is the kingdom of heaven actually going to be as perfect as Christians imagine it? Is the lion really going to lie down with lamb? Is God really all-loving and omnipotent? Is that beam of light really infinite? That’s not really the point.

For believers (which I do not count myself among), leaving aside the beam of light, that very much is the point. That God really is up there/down here/in here and it is our duty to live as He has shown us. "He hath shewed thee, O man, what is good; and what doth the LORD require of thee, but to do justly, and to love mercy, and to walk humbly with thy God?" (Micah 6:8.)

Another has commented that the Unbendable Arm demo is headology, and that you can just do the thing without tricking yourself into it.

I must wonder whether this "as if" conception of God will wear thin in adversity. Are you really going to put a life's work into building a cathedral for the glory of God on that foundation?

I remember that back in the 60's, the hippy era, the concept arose of the "earth mother", a woman fulfilling an ideal of bountifulness, nurturing, and attunement to Mother Nature. It did not take long for people — or at least, the women — to realise that this was a scam to get the women to do all the cooking and provide free sex (which was called free love, but that is another story). As a woman of the time put it, "there's only so many pounds of carrots you can scrub and still imagine you're having a valid spiritual experience."

Why should I bargain for a portion of pie if I can just take whatever I want? This is the real game between an ASI and humanity:

When the cashier smiles at you 1% more than usual, you probably don't stop and wonder whether it's a sign or not. You won't think anything of it because it's well within the noise -- but you might smile 1% more in return without noticing that you do. She might smile an additional 1% the next time, and you might respond in kind. Before you know it people might be saying "Get a room, you two!".

I do not believe that any such frog-boiling has ever happened to me.

It is said that humans who are not paying attention are not general intelligences. I try to cultivate the virtue of attention.

The distinction always exists.

The perception and the perception of the perception are always different things. But to return to the situation at hand, I find it difficult to imagine responding to subtle clues by asking for the other person's phone number, without being aware that "here I am, responding to what I think are probably clues by considering possible responses", and also going on to higher levels of the ladder, like considering whether my perception of these supposed clues is correct, how i know what I think I know, deciding whether if clues these be I wish to build something on them (depending on the person, maybe I'd rather put them off than draw them on), deciding how long to maintain plausible deniability that there is anything going on here and when to break cover, and so on. The searchlight of attention, of noticing, sweeps over all of these, in less time than it takes to read this paragraph.

At least, that is how it is for me.

Higher-level posts are normally filled by promoting those who entered at a lower level. When there are no lower level staff, what replaces that process?