Don't focus on internal knowledge vs black-box prediction, instead think of model complexity and how big our constructed model has to be in order to predict correctly.
A human may be its own best model, meaning that perfect prediction requires a model at least as complex as the thing itself. Or the internals may contain a bunch of redundancy and inefficiency, in which case it's possible to create a perfect model of behavior and interaction that's smaller than the human itself.
If we build the predictive model from sufficient observation and black-box techniques, we might be able to build a smaller model that is perfectly representative, or we might not. If we build it solely from internal observation and replication, we're only ever going to get down to the same complexity as the original.
I include hybrid approaches (use internal and external observations to build models that don't operate identically to the original mechanisms) in the first category: that's still black-box thinking - use all info to model input/output without blindly following internal structure.
This seems correct to me. Thank you.
To the extent that an agent is predictable, it must be:
The first implies that the predictor has collected data emitted by the agent.
The second implies that the agent has internal structure and that the predictor has the capacity to represent the internal structure of the other agent.
In general, we can say that people do not have the capacity to explicitly represent other people very well. People are unpredictable to each other. This is what makes us free. When somebody is utterly predictable to us, their rigidity is a sign of weakness or stupidity. They are following a simple algorithm.
We are able to model the internal structure of worms with available computing power.
As we build more and more powerful predictive systems, we can ask: is our internal structure in principle knowable by this powerful machine?
(x-posted to digifesto)