Part of the reason why the sentence bothers me is that I'm a mathematician and it wasn't obvious to me that there is a useful way of making the statement mathematically precise.
Is it perhaps that my wording appears to be implying that I meant more than "goals can be arranged in a graph of interdependent nodes that recursively update one another for weighting"?
So this is a little better and that may be part of it. Unfortunately, it isn't completely obvious that this is true either. This is a property that we want goal systems to have in some form. It isn't obvious that all goal systems in some broad sense will necessarily do so.
It isn't obvious that all goal systems in some broad sense will necessarily do so.
"All" goal systems don't have to; only some. The words I use to form this sentence do not comprise the whole of the available words of the English language -- just the ones that are "interesting" to this sentence.
It would seem implicit that any computationally-based artificial intelligence would have a framework for computing. If that AI has volition, then it has goals. As we're already discussing, topically, a recursively improving AI, then it has voli...
I have stopped understanding why these quotes are correct. Help!
More specifically, if you design an AI using "shallow insights" without an explicit goal-directed architecture - some program that "just happens" to make intelligent decisions that can be viewed by us as fulfilling certain goals - then it has no particular reason to stabilize its goals. Isn't that anthropomorphizing? We humans don't exhibit a lot of goal-directed behavior, but we do have a verbal concept of "goals", so the verbal phantom of "figuring out our true goals" sounds meaningful to us. But why would AIs behave the same way if they don't think verbally? It looks more likely to me that an AI that acts semi-haphazardly may well continue doing so even after amassing a lot of computing power. Or is there some more compelling argument that I'm missing?