I noted that. I have little doubt (approx 0 doubt) about your ability to understand the fallacy. I'm thinking this may be the point where you made the mistake and then that idea got so deeply embedded that the maths you've derived from this point in the sequence as a description of what goes on is where you're relying a bit too heavily on the maths to make predictions of AI trajectories. 

I say this with a ton of humility knowing the limits of my knowledge, but it does feel like I'm right. To me.

It feels to me like the terminal function of any intelligence that's generalisable, is self-preservation BEFORE any optimisation. Of course, I'm thinking about biological substrate not silicon. 

(PS. I discovered the Zizians only a couple of weeks ago when you posted. So sorry you're getting your name dragged. Happy to help if I can). 

(PPS. I don't believe in information hazards. I can conceptualise the notion, but if there is any takeaway I had from Jaynes and the Mind Projection Fallacy in particular (limited math understanding, can't derive a lot of his math, but feel strongly enough that I understand the main ideas conceptually at least), is that I now think every problem is an information problem. Is this where I may be getting it wrong? (According to you) ) 

I think this is the exact post where Yudkowsky goes from being perfectly logical about everything and commits the Mind Projection Fallacy. 

Paradoxes exist only in the mind.

(PS. At the risk of stating the obvious, I fully reserve the right to change my mind)

He didn't "fail". You'll are just talking about different things.

Congratulations. Now I'm going to read it. 

to deceive you into believing that it is a 𝘝-maximizer.

If it is smart enough to know that it should tell you it's a V-maximizer, then it's smart enough to know that you wanted a V-maximizer.

I would expect people to react mainly to the part about the IMO gold medalist, even though the base rate for being an IMO gold medalist is higher than the base rate for authoring the most-reviewed Harry Potter fanfiction.

This is true, but the reaction is for the conjunction of IMO Gold Medalist and moset-reviewed HP fanfic. 

I want to upvote this again.

Well.. I don't think the process is too rigid. You can always discuss it in advance. Also, there are a few things that you do know are better for you, but are still not able to achieve. But yes, there is a risk. I do not think the risk is so great as to not even give this a try though.

Besides, we don't even know if this works yet!

It is. Judgment comes before.

I'm only suggesting this as a trick, once you've already figured out what it is that you need to do. I suppose I could offer my own feedback, but I was hoping that I would at least try and see if it worked over a larger sample space.

Thanks for the input!

I'm not able to correct the hyperlink part, but I did change the name to Phil Goetz as was due.