That is meant to be informative to those wondering what Holden was talking about. I do not know what do you mean by 'some substance'.
edit: Actually, okay, I should be less negative. It probably is a case of accidental/honest self deception here, and the technobabbling arose by honest attempt to best communicate the intuitions. You approach problem from direction - how do we make a safe oracle out of some AGI model that runs in your imagination reusing the animalism to predict it. Well you can't. That's quite true! However, the actual software is using branches and loops and arithmetic, it's not run on animalism module of your brain, it's run on computer. There's this utility function, and there's the solver which finds maximum of it (which it just does, it's not trying to maximize yet another utility ad infinitum, don't model it using animalism please), and together they can work like animalist model, but the solver does NOT work as animalist model does.
edit: apparently animalism is not exactly the word I want. The point is, we have a module in our brain made for predicting other agents of a very specific type (mammals) and it is a source of some of the intuitions about the AI.
former post:
Ultimately, I figured out what is going on here. Eliezer and Luke are two rather smart technobabblers. That really is all to it. Not explaining anything about the technobabble. Not worth it. The technobabble is what results when one is thinking in terms of communication-level concepts rather than in terms of reasoning-level concepts, within a technical field. The technobabble can and has been successfully published in peer reviewed journals ( Bogdanov affair ) and, sadly, can even be used to acquire a PhD.
The Oracle AI as defined here is another thing defined in terms of the 'utility' as known on LW and has nothing to do with the solver component of the agents as currently implemented.
The bit about predictor having implicit goal to make world match predictions is utter and complete bullshit not even worth addressing. It arises from thinking in terms of communication level concepts such as 'should' and 'want', which can be used to talk about AI but can not be used to reason about AI.
While I understand your frustration, in my experience, you will probably get better results on this forum with reason rather than emotion.
In particular, when you say
Eliezer and Luke are two rather smart technobabblers.
you probably mean something different from what is used on Sci-fi shows.
Similarly,
The implicit goal about world having to match predictions is utter and complete bullshit not even worth addressing.
comes across as a rant, and so is unlikely to convince your reader of anything.
Presently, the 'utility maximizers' work as following: given a mathematical function f(x) , a solver finds the x that corresponds to a maximum (or, typically, minimum) of f(x) . The x is usually a vector describing the action of the agent, the f is a mathematically defined function which may e.g. simulate some world evolution and compute the expected worth of end state, given action x, as in f(x)=h(g(x)) where h computes worth of world state g(x), and g computes the world state at some future time assuming that action x was taken.
For instance, the f may represent some metric of risk, discomfort, and time, over a path chosen by a self driving car, in a driving simulator (which is not reductionist). In this case this metric (which is always non-negative) is to be minimized.
In a very trivial case, such as finding the cannon elevation at which the cannonball will land closest to the target, in vacuum, the solution can be found analytically.
In more complex cases multitude of methods are typically employed, combining iteration of potential solutions with analytical and iterative solving for local maximum or minimum. If this is combined with sensors and the model-updater, and actuators, an agent like a self driving car can be made.
Those are the utility functions as used in the field of artificial intelligence.
A system can be strongly superhuman at finding maximums to functions, and ultimately can be very general purpose, allowing it's use to build models which are efficiently invertible into a solution. However it must be understood that the intelligent component finds mathematical solutions to, ultimately, mathematical relations.
The utility functions as known and discussed on LW seem entirely different in nature. Them are defined on the real word, using natural language that conveys intent, and seem to be a rather ill defined concept for which the bottom-up formal definition may not even exist. The implementation of such concept, if at all possible, would seem to require a major breakthrough in the philosophy of mind.
This is an explanation of an important technical distinction mentioned in Holden Karnofsky's post.
On the discussion in general: It may well be the case that it is very difficult or impossible to define a system such as self driving car in terms of the concepts that are used on LW to talk about intelligences. In particular, the LW's notion of "utility" does not seem to allow to accurately describe the kind of tool that Holden Karnofsky was speaking of, in terms of this utility.