Saying that there is an agent refers (in my view; definition for this thread) to a situation where future events are in some sense expected to be optimized according to some goals, to the extent certain other events ("actions") control those future events. There might be many sufficient conditions for that in terms of particular AI designs, but they should amount to this expectation.
So an agent is already associated with goals in terms of its actual effect on its environment. Given that agent's own future state (design) is an easily controlled part of the environment, it's one of the things that'll be optimized, and given that agents are particularly powerful incantations, it's a good bet that future will retain agent-y patterns, at least for a start. If future agent has goals different from the original, this by the same definition says that the future will be optimized for different goals, and yet in a way controllable by original agent's actions (through the future agent). This contradicts that the original agent is an agent (with original goals). And since the task of constructing future agent includes specification of goals, original agent needs to figure out what they are.
And since the task of constructing future agent includes specification of goals . . .
There seems to be a leap, here. An agent, qua agent, has goals. But is it clear that the historical way in which the future-agent is constructed by the original agent must pass through an explicit specification of the future-agent's goals? The future-agent could be constructed that way, but must it? (Analogously, a composite integer has factors, but a composite can be constructed without explicitly specifying its factors.)
I have stopped understanding why these quotes are correct. Help!
More specifically, if you design an AI using "shallow insights" without an explicit goal-directed architecture - some program that "just happens" to make intelligent decisions that can be viewed by us as fulfilling certain goals - then it has no particular reason to stabilize its goals. Isn't that anthropomorphizing? We humans don't exhibit a lot of goal-directed behavior, but we do have a verbal concept of "goals", so the verbal phantom of "figuring out our true goals" sounds meaningful to us. But why would AIs behave the same way if they don't think verbally? It looks more likely to me that an AI that acts semi-haphazardly may well continue doing so even after amassing a lot of computing power. Or is there some more compelling argument that I'm missing?