Scientist by training, coder by previous session,philosopher by inclination, musician against public demand.
I'm specifically addressing the argument for a high probability of near extinction (doom) from AI...
Eliezer Yudkowsky: "Many researchers steeped in these issues, including myself, expect that the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die. "
....not whether it is barely possible, or whether other, less bad outcomes (dystopias) are probable. I'm coming from the centre, not the other extreme
Doom, complete or almost complete extinction of humanity, requires a less than superintelligent AI to become superintelligent either very fast , or very surreptitiously ... even though it is starting from a point where it does not have the resources to do either.
The "very fast" version is foom doom...Foom is rapid recursive self improvement (FOOM is supposed to represent a nuclear explosion)
The classic Foom Doom argument (https://www.greaterwrong.com/posts/kgb58RL88YChkkBNf/the-problem) involves an agentive AI that quickly becomes powerful through recursive self improvement, and has a value/goal system that is unfriendly and incorrigible.
The complete argument for Foom Doom is that:-
The AI will have goals/values in the first place (it wont be a passive tool like GPT*),.
The values will be misaligned, however subtly, to be unfavorable to humanity.
That the misalignment cannot be detected or corrected.
That the AI can achieve value stability under self modification.
That the AI will self modify in way too fast to stop.
That most misaligned values in the resulting ASI are highly dangerous (even goals that aren't directly inimical to humans can be a problem for humans, because the AS I might want to director sources away from humans.
And that the AI will have extensive opportunities to wreak havoc: biological warfare (custom DNA can be ordered by email), crashing economic systems (trading can be done online), taking over weapon systems, weaponing other technology and so on.
It’s a conjunction of six or seven claims, not just one. ( I say "complete argument " because pro doomers almost always leave out some stages. I am not convinced that rapid self improvement and incorrigibility are both needed, both needed, but I am sure that one or the other is. Doomers need to reject the idea that misalignment can be fixed gradually, as you go along. . A very fast-growing ASI, foom, is way of doing that; and assumption that AI's will resist having their goals changed is another).
Obviously the problem is that to claim a high overall probability of doom, each claim in the chain needs to have a high probability. It is not enough for some of the stages to be highly probable, all must be.
There are some specific weak points.
Goal stability under self improvement is not a given: it is not possessed by all mental architectures, and may not be possessed by any, since noone knows how to engineer it, and humans appear not to have it.
The Orthogonality Thesis (https://www.lesswrong.com/w/orthogonality-thesis)is sometimes mistakenly called on to support to support goal stability. It implies that a lot of combinations of goals and intelligence levels are possible, but doesn't imply that all possible minds have goals, or that all goal driven agents have fixed, incorrigible goals. There are goalless and corrigible agents in mindspace, too. That's not just an abstract possibility. At the time of writing, 2025, our most advanced AI's, the Large Language Models, are non agentive and corrigible.
It is plausible that an agent would desire to preserve its goals, but the desire to preserve goals does not imply the ability to preserve goals. Therefore, no goal stable system of any complexity exists on this planet, and goal instability cannot be assumed as a default or given. So the orthogonality thesis is true of momentary combinations of goal and intelligence, given the provisos above, but not necessarily true of stable combinations.
Another thing that doesn't prove incorrigibility or goal stability is von Neumann rationality. Frequently appealed to in MIRI 's early writings , it is an idealised framework for thinking about rationality , that doesn't app!y to humans, and therefore doesn't have to apply to any given mind.
There are arguments that AI's will become agentive because that"s what humans want. Gwerns Branwen's confusingly titled "Why Tool AIs Want to Be Agent AIs" ( https://gwern.net/tool-ai) is an example. This is true, but in more than one sense:-
The basic idea is that humans want agentive AI's because they are more powerful. And people want power, but not at the expense of control. Power that you can't control is no good to you. Taking the brakes off a car makes it more powerful, but more likely to kill you. No army wants a weapon that will kill their own soldiers, no financial organisation wants a trading system that makes money for someone else, or gives it away to charity, or causes stick market crashes. The maximum amount of power and the minimum of control is an explosion.
One needs to look askance at what "agent" means as well. Among other things, it means an entity that acts on behalf of a human -- as in principal/agent.(https://en.m.wikipedia.org/wiki/Principal–agent_problem) An agent is no good to its principal unless it has a good enough idea of its principal's goals. So while people will want agents, they wont want misaligned ones -- misalgined with themselves, that is. Like the Orthogonality Thesis, the argument is not entirely bad news.
Of course, evil governments and corporations controlling obedient superintelligences isn't a particularly optimistic scenario, but it's dystopia, not doom.
Yudkowsky's much repeated argument that safe , well-aligned behaviour is a small target to hit ... could actually be two arguments.
One would be the random potshot version of the Orthogonality Thesis, where there is an even chance of hitting any mind, and therefore a high chance ideas of hitting an eldritch, alien mind. But equiprobability is only one way of turning possibilities into probabilities, and not particularly realistic. Random potshots aren't analogous to the probability density for action of building a certain type of AI, without knowing much about what it would be.
While, many of the minds in mindpsace are indeed weird and unfriendly to humans, that does not make it likely that the AIs we will construct will be. we are deliberately seeking to build certainties of mind for one thing, and have certain limitations, for another. Current LLM 's are trained in vast copora of human generated content, and inevitably pick up a version of human values from them.
Another interpretation of the Small Target Argument is, again , based on incorrigibility. Corrigibility means you can tweak an AI's goals gradually, as you go on, so there s no need to get them exactly right on the first try.
By far the best definition I’ve ever heard of the supernatural is Richard Carrier’s: A “supernatural” explanation appeals to ontologically basic mental things, mental entities that cannot be reduced to nonmental entities.
Physicalism, materialism, empiricism, and reductionism are clearly similar ideas, but not identical. Carrier's criterion captures something about a supernatural ontology, but nothing about supernatural epistemology. Surely the central claim of natural epistemology is that you have to look...you can't rely on faith , or clear ideas implanted in our minds by God.
it seems that we have very good grounds for excluding supernatural explanations a priori
But making reductionism aprioristic arguably makes it less scientific...at least, what you gain in scientific ontology, you lose in scientific epistemology.
I mean, what would the universe look like if reductionism were false
We wouldn't have reductive explanations of some apparently high level phenomena ... Which we don't.
I previously defined the reductionist thesis as follows: human minds create multi-level models of reality in which high-level patterns and low-level patterns are separately and explicitly represented. A physicist knows Newton’s equation for gravity, Einstein’s equation for gravity, and the derivation of the former as a low-speed approximation of the latter. But these three separate mental representations, are only a convenience of human cognition. It is not that reality itself has an Einstein equation that governs at high speeds, a Newton equation that governs at low speeds, and a “bridging law” that smooths the interface. Reality itself has only a single level, Einsteinian gravity. It is only the Mind Projection Fallacy that makes some people talk as if the higher levels could have a separate existence—different levels of organization can have separate representations in human maps, but the territory itself is a single unified low-level mathematical object. Suppose this were wrong.
Suppose that the Mind Projection Fallacy was not a fallacy, but simply true.
Note that there are four possibilities here...
I assume a one level universe, all further details are correct.
I assume a one level universe, some details may be incorrect
I assume a multi level universe, all further details are correct.
I assume a multi level universe, some details may be incorrect.
How do we know that the MPF is actually fallacious, and what does it mean anyway?
If all forms of mind projection projection are wrong, then reductive physicalism is wrong, because quarks, or whatever is ultimately real, should not be mind projected, either.
If no higher level concept should be mind projected, then reducible higher level concepts shouldn't be ...which is not EY's intention.
Well, maybe irreducible high level concepts are the ones that shouldn't be mind projected.
That certainly amounts to disbelieving in non reductionism...but it doesn't have much to do with mind projection. If some examples of mind projection are acceptable , and the unacceptable ones coincide with the ones forbidden by reductivism, then MPF is being used as a Trojan horse for reductionism.
And if reductionism is an obvious truth , it could have stood on its own as apriori truth.
Suppose that a 747 had a fundamental physical existence apart from the quarks making up the 747. What experimental observations would you expect to make, if you found yourself in such a universe?
Science isn't 100% observation,it's a mixture of observation and explanation.
A reductionist ontology is a one level universe: the evidence for it is the success of reductive explanation , the ability to explain higher level phenomena entirely in terms of lower level behaviour. And the existence of explanations is aposteriori, without being observational data, in the usual sense. Explanations are abductive,not inductive or deductive.
As before, you should expect to be able to make reductive explanations of all high level phenomena in a one level universe....if you are sufficiently intelligent. It's like the Laplace's Demon illustration of determinism,only "vertical". If you find yourself unable to make reductive explanations of all phenomena, that might be because you lack the intelligence , or because you are in a non reductive multi level universe or because you haven't had enough time...
Either way, it's doubtful and aposteriori, not certain and apriori.
If you can’t come up with a good answer to that, it’s not observation that’s ruling out “non-reductionist” beliefs, but a priori logical incoherence"
I think I have answered that. I don't need observations to rule it out. Observations-rule it-in, and incoherence-rules-it-out aren't the only options.
People who live in reductionist universes cannot concretely envision non-reductionist universes.
Which is a funny thing to say, since science was non-reductionist till about 100 years ago.
One of the clinching arguments for reductionism.was the Schrödinger equation, which showed that in principle, the whole of chemistry is reducible to physics, while the rise of milecular biology showeds th rreducxibility of Before that, educators would point to the de facto hierarchy of the sciences -- physics, chemistry, biology, psychology, sociology -- as evidence of a multi-layer reality.
Unless the point is about "concretely". What does it mean to concretely envision a reductionist universe? Pehaps it means you imagine all the prima facie layers, and also reductive explanations linking them. But then the non-reductionist universe would require less envisioning, because byit's the same thing without the bridging explanations! Or maybe it means just envisioing huge arrays of quarks. Which you can't do. The reductionist world view , in combination with the limitations of the brain, implies that you pretty much have to use higher level, summarised concepts...and that they are not necessarily wrong.
But now we get to the dilemma: if the staid conventional normal boring understanding of physics and the brain is correct, there’s no way in principle that a human being can concretely envision, and derive testable experimental predictions about, an alternate universe in which things are irreducibly mental. Because, if the boring old normal model is correct, your brain is made of quarks, and so your brain will only be able to envision and concretely predict things that can predicted by quarks.
"Your brain is made of quarks" is aposteriori, not apriori.
Your brain being made of quarks doesn't imply anything about computability. In fact, the computatbolity of the ultimately correct version of quantum physics is an open question.
Incomputability isn't the only thing that implies irreducibility, as @ChronoDas points out.
Non reductionism is conceivable, or there would be no need to argue for reductionism.
The Deutsch-Yudkowsky argument for the Many Worlds Interpretation states that you can take the core of Quantum Mechanics -- the Schrödinger wave equation, and the projection postulate -- remove the projection postulate (also known as collapse and reduction ), and end with a simpler theory that is still adequate to explain observation. The idea is that entanglement can replace collapse: a scientist observing a superposed state becomes becomes entangled with it, an effectively splits into two, each having made a definite observation.
Moreover Yudkowsky, following David Deutsch, holds the many worlds interpretation to be obviously correct, in contrast to the majority of philosophers and physicists, who regard the problem of interpreting QM as difficult and unsolved.
This has some problems.
(Which are to do with the specific argument, and the level of certainty ascribed to it. To say that you cannot be certain about a claim is not to say it is false. To point out that one argument for a claim does not work is likewise not to say that the claim itself is false. There could be better arguments for these versions of many worlds, or better many worlds theories, for that matter).
The first thing to note is that there is more than one quantum mechanical many worlds theory. What splittng is...how complete and irrevocable it is ... varies between particular theories. So does the rate of splitting, so does the mechanism of splitting.
The second thing to note is that many worlders are pointing at something implied the physical formalism and saying "that's a world"....but whether it qualifies as a world is a separate question from whether it's in the formalism , and a separate kind of question, from whether it is really there in the formalism. One would expect a world, or universe, to be large, stable, non-interacting, and so on. It's possible to have a n interpretation without collapse or worlds. A successful MWI needs to jump three hurdles: empirical correctness, mathematical correctness and conceptual correctness -- actually having worlds
The third problem to note is that all outstanding issues with MWI are connected in some way with quantum mechanical basis....a subject about which Deutsch and Yudkowsky have little to say.
There is an approach to MWI based on coherent superpositions, and a version based on decoherence. These are (for all practical purposes) incompatible opposites, but are treated as interchangeable in Yudkowsky's writings.
Quantum superposition is a fundamental principle of quantum mechanics that states that linear combinations of solutions to the Schrödinger equation are also solutions of the Schrödinger equation. This follows from the fact that the Schrödinger equation is a linear differential equation in time and position. (WP)
Coherent superpositions are straightforwardly implied by the core mathematics of Quantum mechanics. They are small scale in two senses: they can go down to the single particle level, and it is difficult to.maintain large coherent superpositions even if you want to. They are also possibly observer dependent, reversible, and continue to interact (strictly speaking , interfere) after "splitting". The last point is particularly problematical. because if large scale coherent superposition exist , that would create naked eye, macrocsopic evidence:, e.g. ghostly traces of a world where the Nazis won. All in all, a coherent superposition isn't a world you could live in.
I said complex coherent superpositions are difficult to maintain. What destroys them? Environmental induced decoherence!
Interference phenomena are a well-known and crucial aspect of quantum mechanics, famously exemplified by the two-slit experiment. There are many situations, however, in which interference effects are artificially or spontaneously suppressed. The theory of decoherence is precisely the study of such situations. (SEP)
Decoherence tries to explain why we don't notice "quantum weirdness" in everyday life -- why the world of our experience is a more-or-less classical world. From the standpoint of decoherence, sure there might not be any objective fact about which slit an electron went through, but there is an objective fact about what you ate for breakfast this morning: the two situations are not the same!
The basic idea is that, as soon as the information encoded in a quantum state "leaks out" into the external world, that state will look locally like a classical state. In other words, as far as a local observer is concerned, there's no difference between a classical bit and a qubit that's become hopelessly entangled with the rest of the universe.
(http://scottaaronson.com/democritus)
Decoherence is the study of interactions between a quantum system (generally a very small number of microscopic particles like electrons, photons, atoms, molecules, etc. - often just a single particle) and the larger macroscopic environment, which is normally treated "classically," that is, by ignoring quantum effects, but which decoherence theorists study quantum mechanically. Decoherence theorists attribute the absence of macroscopic quantum effects like interference (which is a coherent process) to interactions between a quantum system and the larger macroscopic environment.(www.informationphilosopher.com)
Decoherent branches are necessarily large, since decoherence is a high level phenomenon. They are also stable, non interacting and irreversible...everything that would be intuitively expected of a "world". But there is no empirical evidence for them (in the plural) , nor are they obviously supported by the core mathematics of quantum mechanics, the Schrödinger equation.
We have evidence of small scale coherent superposition, since a number of observed quantum effects depend on it, and we have evidence of decoherence, since complex superposition are difficult to maintain. What we don't have evidence of is decoherence into multiple branches. From the theoretical perspective, decoherence is a complex , entropy like process which occurs when a complex system interacts with its environment. But without decoherence, MW doesn't match observation. So there is no theory of MW that is both simple and empirically adequate, contra Yudkowsky and Deutsch.
The original, Everettian, approach is based on coherence. (Yudkowsky says "Macroscopic decoherence, a.k.a. many-worlds, was first proposed in a 1957 paper by Hugh Everett III" ... but the paper doesn't mention decoherence [1] ) As such, it fails to predict classical observations -- at all -- it fails to predict the appearance of a broadly classical universe. If everything is coherently superposed, so are observers...but the naturally expected experience an observer in coherent superposition with themselves, is that they function as a single observer making ambiguous, superposed observations ... not two observers each making an unambiguous , classical observation, and each unaware of the other. Such observers would only ever see superpositions of dead and living cats, etc.
(A popular but mistaken idea is that full splitting happens microscopically, at every elementary interaction But that would make complex superpositions non-existent, whereas a number of instruments and technologies depend on them -- so it's empirically false).
Later, post 1970s, many world theorists started to include decoherence to make the theory more empirically adequate, but inasmuch as it is additional structure, it places the simplicity of MWI in doubt. In the worst case, the complexity is SWE+decoherence+preferred basis, whereas in the best case, it's SWE alone, because decoherence is implicit in SWE, and preferred basis is implicit in decoherence. Decoherentists hope to show that the theory can be reduced to core QM, such as the Schrödinger equation, but it currently uses more complex math, the "reduced density matrix". The fact that this research is ongoing is strong evidence that the whole problem was not resolved by Everetts's 1957 paper. In any case, without a single definitive mechanism of decoherence, there is no definitive answer to "how complex is MWI".
And single-universe decoherence is quite feasible. Decoherence adds something to many worlds, but many worlds doesn't add anything to decoherence.
So, coherent superpositions exist, but their components aren't worlds in any intuitive sense; and decoherent branches would be worlds in the intuitive sense, but decoherence isn't simple. Also, theoretically and observationally, decoherence could be a single world phenomenon. Those facts -- the fact that it doesn't necessarily involve multi way branching, and the fact that it is hard to evaluate its complexity because there is not a single satisfactory theory for it -- means it is not a "slam dunk" in Yudkowsky's sense.
The Yudkowsky-Deutsch claim is that there is a single MW theory, which explains everything that needed explaining, and is obviously simpler than its rivals. But coherence doesn't save appearances , and decoherence, while more workable, is not known to be simple. So neither theory has both virtues
Which makes the term *Everett branch" rather confusing. The writer possibly means a decohered branch, under the mistaken assumption that Everett was talking about them. Everett's dissertation can be found here ↩︎
My reason to believe that consciousness is fundamental rather than structural is that it seems to me more Occamian to believe that consciousness is eternal
A single non physical consciousness might be more Occamian than multiple ones, but it's less Occamian than a purely physical ontology.
ETA
The AI is already not aligned
For what value of "aligned"? The current state of play is thanking just about anything could be said to be aligned or misaligned, sending in exactly what the speaker means by the terms.
There is a lot of semantic confusion between people who use "alignment" in an engineering sense,meaning something that renders current AI safe in good enough way -- and the people who use it to mean a maths style solution , that applies perfectly to every case. A completely unaligned AI would be completely unco-operative , and therefore of no commercial use, so the prevailing level of alignment isn't zero.
You even acknowledge that there are different kinds of alignment here:-
We could build an ASI that is aligned, but not aligned with humanity as a whole (whatever that means), s
But even if we didn’t, and everything seemed fine, I would not believe LLMs are aligned, because value is complex and fragile.
That need not be an argument against good enough alignment.
One way alignment can be made to look difficult is stating that it has to be done in a maximal way to achieve minimal results. The minimal result is not killing us, all, the maximal way is instilling into the AI every nuance of human value including aesthetic value. Under circumstances when a powerful ASI takes over and starts running things according to its original programming, without listening to feedback, a detailed knowledge of human values would be necessary to create a utopia. But that is a far cry from not killing us all.
Another way of making successful alignment look difficult is the making the assumption that AI's will become very powerful , very quickly, in an unsupervised way, so that humans only have one chance to get alignment correct, before the ASI becomes too powerful to listen to humans. That's the idea underlying the Fragility of Value. It doesn't actually matter how complex or subtle value is , so long as you can tweak a specification of value at leisure. If course, a fast "take off" isn't impossible, it is too often treated as a certainty, and too often left as an implicit assumption
LLMs are agents. They’re remarkably non-agentic
That looks another semantic confusion. If they are remarkably non-agentic, why not round them down to non-agents?
And notice the number of things that have to go right in order for this stage to be where doom stops:
- The AI has to be scary
- And we have to notice it being scary
- And we have to band together to try and stop it
- AND we have to win
- AND after the victory we have to ~permanently ban this fearsome technology that already exists
To quote a thinker that I respect: “If they each have an 80% chance
The negation of a conjunction is a disjunction.
https://en.wikipedia.org/wiki/De_Morgan's_laws
The argument for Doom is mostly or wholly conjunctive, so the argument for no-doom is mostly or wholly disjunctive. If ASI is impossible, no doom, if ASI is not placed in charge of everything, no doom, etc.
To put it a other way , if A is low probability, not-A can't also be.
The multi stage fallacy is not a fallacy in the sense that conjunctive arguments are never unlikely: rather there can be, burning have to be pitfalls in multi stage arguments. You are still unlikely to to win the lottery and be struck by lightning.
Harris can also be critiqued from the direction of libertarian free will.
Half of the answer is to recognise that my unconscious mind is still mine -- following its prompting is not being under the control of someone else. The other half us understanding that the conscious mind can still exert control.
The basic mechanism is that the unconscious mind proposes various ideas and actions , which the conscious mind decides between, and finally puts the chosen one into action. (Since you only have one body, it is important to decide on a clear course of action, not half of one thing and half of another).
Sam Harris makes much of the fact that the conscious mind, the executive function, does not predetermine the suggestions, and concludes that in the absence of of predetermination, there is no conscious control at all. In contrast, I argue that the choice between impulses, the decision to act on one rather than another, the gatekeeping mechanism, is conscious control -- and conscious control clearly exists in health adults.
If there is indeterminism the mechanism (and there doesn't have to be) it provides the libertarian could-have-done-otherwise as well as conscious control.
but there’s obviously no way Sam would say something like “consciousness is causally disconnected from the rest of the universe, and can never influence future decision making processes in the brain.”
You mean, no way he would embrace metaphysical epiphenomenalism, because he is a physicalist?
Actually, it's not clear that he is a physicalist. It's possible to be a physical epiphenomenalist , regarding consciousness as a brain module with no downward casual affect, but it's strange , because it implies something complex, but with no function evolved.
Retributive justice—as in really within jurisprudence or so—questions are essentialy when we discuss free will, and here’s where most people are stupidly confused while SH’s exposition and interpretation is spot on about it.
But it's not clear:-
• Whether retributive punishment is a US or global phenomenon
• Whether it's caused by fee will beliefs.
• Whether there is a radically, rather incrementally, different approach to punishment.
Is it a Uniquely US Phenomenon?
It might be the case that US style prisons are like that because of a desire to inflict suffering because of a belief in free will...but many other explanations are possible. Most of the issues explained by simply not funding prisons well. After all, being in the same environment as a bunch of criminals is pretty intrinsic, not some special punishment. If the state was spending money on torture equipment, then you'd have evidence that they were making a special effort to cause suffering, rather than just doing things on the cheap.
Prisons in poor countries are invariably awful: no special effort is required to induce suffering. The harshness of US prison system is not explained by poverty ,since the US is the world's richest large country, but does not have to be explained by free will. One distinctive factor in the US is the democratisation of the criminal justice system. Public prosecutors are elected, and therefore need a high profile: committing to slamming people up for long periods is apparently more attention-grabbing than releasing the innocent
Is It Caused by Religion?
Harris, Sapolsky and their supporters seem to like the liberal Scandinavian approach. But Scandinavia is not particularly atheist. For instance,Norway had a state religion until 2012, and 70% of the population are Lutheran, a sect that upholds free will. So theism doesn't simply predict a punitive criminal justice syste
Soviet Russia, by contrast, was officially atheist..and materialistic and deterministic ... yet had a very harsh penal system. So atheism doesn't simply predict a gentle criminal justice system.
As far as I can see, the main predictors of a humane penal.system are a combination of societal wealth and political liberalism. But philosophical beliefs in theism and atheism, free will or determinism, are not strongly correlated with wealth or liberalism.
Is There an Alternative?
If you feel that someone is a danger to the community, then putting them in jail has a justification different from punishing them for their sins. But you might want to punish them by jailing them if you believe in free will, as well...so you cant infer a fundamental philosophical difference from the fact that that some people are in jail. Even if you want to rehabilitate them you still have to make them turn up to therapy sessions when they don't want
And punishment is behaviour shaping,therefore therapeutic, in some ways.
In figuring out what would constitute good conduct and productive discourse, it’s important to appreciate how bizarre the human practice of “discourse” looks in light of Aumann’s dangerous idea.
Within the rationalsphere , Aumanns Theorem is simplified into something like "reasonable people can't agree to differ". Yet they can, because the conditions on AAT are much more stringent than an informal understanding of what it is to be reasonable. So AAT lacks real world applicability..
There is a further problem that prior beliefs can include beliefs about epistemology, about what constitutes evidence.
Controversies that are sufficiently deep, or which cut across cultural boundaries run into a problem where, not only do parties disagree about the object level issue, they also disagree about underlying questions of what constitutes truth, proof, evidence, etc. "Satan created the fossils to mislead people" is an example of one side rejecting the other sides evidence as even being evidence . Its a silly example, but there are much more robust ones.
There’s only one reality.
Why do people keep saying that?
It's true that If there were more than one world, that would undercut the ability of different thinkers to come to a uniform conclusion about it, but there is no corollary that existence of a single world would guarantee anything epistemically. It's a necessary condition for convergence, but not sufficient.
If there is one world, that doesn't guarantee that there are agents capable of understanding it, within it ..and if there are intelligent and rational agents, their ability to converge on a single all encompassing truth could be impossible for a number of further reasons. Problems include the inadequacy of epistemic evidence to address all problems, the reliance of logic on axioms (the Munchausen trilemma) , the reliance of epistemology on epistemology (the problem of the criterion), etc.
None of those problems has anything to do with conflicts, dishonesty or different values -- although they undoubtedly exist as well.
If I’m a Bayesian reasoner honestly reporting my beliefs about some question, and you’re also a Bayesian reasoner honestly reporting your beliefs about the same question, we should converge on the same answer, not because we’re cooperating with each other, but because it is the answer.
Bayes has both problems simultaneously. It is dependent on evidence, so it has all the problems of empiricism; and it has the problems of rationalism because it starts with priors. The argument for Bayesianism is that even agents with wildly differing priors can eventually agree, given sufficient evidence. But that is an argument about ideal Bayesians: in reality , the amount of evidence available might be too limited to allow convergence. The ability of realistic Bayesians to formulate hypotheses is also limited. Alice puts most of her credence on the one hypothesis that seems best supported to her, out of the hypotheses she has heard of, or thought up, but Bob might have a better hypothesis that's not in her set.
Real-world disagreements tend to persist; they’re predictable—in flagrant contradiction of how the beliefs of Bayesian reasoners would follow a random walk. From this we can infer that typical human disagreements aren’t “honest”, in the sense that at least one of the participants is behaving as if they have some other goal than getting to the truth.
No, that isn't the inevitable conclusion, because there are so many other sources of disagreement.
That is to say: I do not understand how high-trust, high-cooperation dynamics work. I’ve never seen them. They are utterly outside my experience and beyond my comprehension
I find the black-and-white framing of that very odd. You've spent most of your life in business and academia, which seem to me high co-operation compared to politics and crime, for instance.
What is the butterfly effect mechanism?
If everything is a coherent superposition evolving according to the Schrödinger equation, there is very little cancellation or unwinding of "branches" ..But also no real branching, because superposed states continued to interact.
Decoherence, in the other hand, isn't guaranteed to leave you with more than one branch.
Indeterministic is a loaded word.
In what way? I would have thought "random" is more loaded.
Certainly we don’t believe our actions to be random,
We see them as (up to a point) responsible and controlled, irrespective of fundamental determinism.
but I maintain that the question before compatibilists/semicompatibilists (which I hoped this post would address but IMO doesn’t) is why seeing free will as a human construct is meaningful.
I didn't say free will was a construct. I suggested moral responsibility was.
Of course that is meaningful -- that is a rather low bar -- the question is whether it is sufficient.
So if I create an AI that steals money, I am the greater jerk but the AI is also a jerk?
That doesn't address the argument from usefulness. It is still useful to modify the AI into being a non stealing AI.
Whether an entity has libertarian freewill is another question , that isn't just a matter of social construction, because libertarian free will depends on (in)determinism , which is a fact about the world. But whether LFW is the only possible justification for behaving as if MR exists is also a separate question.
We can be less confused about the issue by treating usefulness and truth as separate issues, and by treating compatibilist free will and libertarian free will as separate issues. There doesn't have to be a single answer to "is free will a social construct?" .
There is clearly something sane, sober adult humans have that toddlers , and drunks dont. Self control doesn't have to be based on fundamental physics. There is a science of control systems that doesn't require the system being controlled to be indeterministic
The point is similar to compatibilism: compatibilism allows something like free will, but as more if a human construct and less of a metaphysical absolute.. This approach, sometimes called semicompatibilism, takes a similar attitude to moral responsibility.
The contrary claim.would be that if we discovered that the world is deterministic,it will would imply no one has ever been morally responsible, and presumably should not have imprisoned or otherwise punished.
"it" isn't a single theory.
The argument that Everettian MW is favoured by Solomonoff induction, is flawed.
If the program running the SWE outputs information about all worlds on a single output tape, they are going to have to be concatenated or interleaved somehow. Which means that to make use of the information, you gave to identify the subset of bits relating to your world. That's extra complexity which isn't accounted for because it's being done by hand, as it were..