Thought before I need to log off for the day,
This line of argument seems to indicate that physical systems can only completely model smaller physical systems (or the trivial model of themselves), and so complete models of reality are intractable.
I am not sure what else you are trying to get at.
The problem seems to be that we have free choice of internal formal systems and
A consistent system, extended by an unprovable axiom, is also consistent, (since if this property was false then we would be able to prove the axiom by taking the extension and searching for a contradiction).
Consequently accepting the unprovable as true or false only has consequences for other unprovable statements.
I don't think this entire exercise says anything.
In short we expect for probablistic logics and decision theories to converge under self-reflection.
I think you might be grossly misreading Godel's incompleteness theorem. Specifically, it proves that a formal system is either incomplete or inconsistent. You have not addressed the possibility that minds are in fact inconsistent/make moves that are symbolically describable but unjustifiable (which generate falsehoods)
We know both happen.
The question then is what to do with inconsistent mind.
Actually, I think this argument demonstrates the probable existence of the opposite of it's top line conclusion.
In short, we can infer from the fact that a symbolic regulator has more possible states than it has inputs that anything that can be modeled as a symbolic regulator has a limited amount of information about it's own state (that is, limited internal visibility). You can do clever things with indexing so that it can have information about any part of its state, but not all at once.
In a dynamic system, this creates something that acts a lot like consciousness, maybe even deserves the name.
How much of a consensus is there on pausing AI
Not much compared to the push to get the stuff that already exists out to full deployment (For various institutions this is meaningful impact on profit margins).
People don't want to fight that, even if they think that further capabilities are bad price/risk/benefits tradeoff.
There is a co-ordination problem where if you ask to pause and people say no, you can't make other asks.
3rd. They might just not mesh/trust that particular movement and the consolidation of platform it represents, and so want to make points on their own instead of joining a bigger organizations demands.
The Good Regulator Premise
Every good regulator of a system must be a model of that system. (Conant and Ashby)
This theorem asserts a necessary correspondence between the regulator (internal representation) and the regulated system (the environment or external system). Explicitly, it means:
A good map (good regulator) of an island (system) is sufficient if external to the island.
But if the map is located on the island, it becomes dramatically more effective if it explicitly shows the location of the map itself ("you are here"), thus explicitly invoking self-reference.
In the case of a sufficiently expressive symbolic system (like the brain), this self-reference leads directly into conditions required for Gödelian incompleteness.Therefore: The brain is evidently a good regulator
Is not the good regulator theorem.
The good regulator theorem is "there is a (deterministic) mapping h:S→R from the states of the system to the states of the regulator."
I don't think this requires embeddedness
An AI control solution is per se a way to control what a AI is doing. If you have AI control, you have the option to tell your AI, don't go FOOM, and have that work.
You would not expect a control measure to continue to work if you told an AI under an AI control protocol to go FOOM.
Improvements in training efficiency are only realized if you actually train the model, and AI control allows you to take the decision to realize training efficiency gains by training a model to a higher point of performance away from the AI that is controlled.
FOOM for software requires that that decision is always yes (either since people keep pushing or the model is in the drivers seat).
So put broadly, the AI control agendas answer to what should you do with an AI system that could go FOOM is not to let it try. Since before it goes FOOM, the model is not able to beat the controls, and going FOOM takes time where the model is working on improving itself not trying hard to not get violently disassembled, an AI control protocol is supposed to be able to turn an AI that goes FOOM over the explicit controls over the course of hours, weeks or months into a deactivated machine.
AI control protocols want to fail loud for this reason. (But a breakout will involving trying to get silent failure for the same reason)
A quick thought on germline engineering, riffing off of https://www.wired.com/story/your-next-job-designer-baby-therapist/, which should not be all that suprising
Even if germline engineering is very good, and so before a child is born we have a lot of control about how things will turn out, once the child is born people will need to change their thinking because they do not have that control any longer. Trying to keep that control for a long time is probably a bad idea. Similarly, if things are not as expected, action as always should be taken on the world as it turned out, not as you planned it. No amount of gene engineering will be so powerful that social factors are completely irrelevant.
The idea of abstraction is generaly a consequential formulation. A thing is an abstraction when its predicts the same consequences as a system produces. Abstraction of moral values would need exactly "moral judgements about an action at different levels of abstraction" to behave properly as collections.
A trivial note
Given standard axioms for Propositional logic
A->A is a tautology
Consequently, 1. Circularity is not a remarkable property (It is not any strong argument for a position)
2. Contradiction still exists
But a system cannot meaningfully say anything about it's axioms other than their logical consequences.
Consequently, since axioms being the logical consequences of themselves is exactly circularity
In a bayesian formulation there is no way of justifying a prior
Or in regular logic you cannot formally justify axioms nor the right to take them.