I think the keyword to my thought process is anthropomorphizing. The intuitive approach to intelligence is that it is a a human characteristic, almost like handsomeness or richness. Hence the pop culture AI is always an anthropomorphic conversation machine from Space Odyssey to Matrix 3 to Knight Rider. For example, it should probably have a sense of humor.

The approach EY/MIRI seems to take is to de-anthropomorphize even human intelligence as an optimization engine. A pretty machine-like thing. This is what I mentioned that I am not sure I can buy it, but willing to accept as a working hypothesis. So the starting position is that intelligence is anthropomorphci, MIRI has a model that de-anthropomorphized it, which is strange, weird, but probably useful, yet at the end we probably need something re-anthropomorphized. Because if not then we don't have AI in the human sense, a conversation machine, we just have a machine that does weird alien stuff pretty efficiently with a rather inscrutable logic.

Looking at humans, beside optimization, the human traits that are considered part of intelligence, such as a sense of humor, or easily understanding difficult ideas in a conversation, are parts of it too, and they lie outside the optimization domain. The outer view is that we can observe intelligent humans optimizing things, this being one of their characteristics, although not exhaustive. However it does not lead to a full understanding of intelligence, just one facet of it, the optimization facet. It is merely an output, outcome of intelligence, not the process but its result.

So when a human with a high IQ tells you to do something in a different way, this is not intelligence, intelligence was the process that resulted in this optimization. To understand the process, you need to look at something else than optimization, the same way to understand software you cannot just look at its output.

What I was asking is how to look at it from the inner view. What is the software on the inside, not what its output are. How does intelligence FEEL like, which may give a clue about how an intelligent software could actually be like as opposed to merely what its outputs (optimization) are. To me a sufficiently challenging task on Raven's Progressive Matrices feels like disassembling a drawing, and then reassembling it as a model that predicts what should be on the missing puzzle. Is that a good approach?

What is AIXI?