Anthropomorphism is the error of attributing distinctly human characteristics to nonhuman processes. As creatures who evolved in a social context, we all have adaptations for making predictions about other humans by empathic inference. When trying to understand the behavior of other humans, it oftentimes is a helpful (and bias-correcting) heuristic to ask, "Well, what would I do in such a situation?" and let that be your prediction. This mode of prediction simply won't do, however, for things (and in this wide universe there are many) that don't share the detailed structure bequeathed on the human brain by evolution, although it is oftentimes tempting... (read more)
| User | Post Title | Wikitag | Pow | When | Vote |
Updateful decision theories change the probability distribution used to evaluate actions over time. Updateless Decision Theory (UDT) does not, instead always maximizing a priori expected utility. This works well in well-defined decision problems with high probability in the prior, but in some senses, does not learn. Open-Minded Updatelessness seeks to combine the advantages of updatelessness and updatefulness by allowing some changes in probabilities, without fully updating on evidence.
Christopher Alexander (1936-2022) was an architect who studied the way nature and traditionally built buildings (such as peasant huts, or cathedrals) are a particular kind of beautiful, and have (he argued) the ability to bring a person back into a sense of perspective (e.g., a person may be quite stressed out about some detail, and then go for a long walk in nature, and find themselves "coming back to themselves.") Alexander attempted to work out a theory of design (for buildings, but also for design work broadly) that would create houses and other built objects with this same sort of beauty and sense of perspective embedded in them. His work inspired the "design patterns" movement in computer science, and, indirectly, wikis.
Evidential Decision Theory – EDT – is a branch of decision theory which advises an agent to take actions which, conditional on it happening, maximizes the chances of the desired outcome. As any branch of decision theory, it prescribes taking the action that maximizes utility, that which utility equals or exceeds the utility of every other option. The utility of each action is measured by the expected utility, the averaged by probabilities sum of the utility of each of its possible results. How
One way to see the actions can influencedifference between evidential utility and causal utility is to contemplate the probabilities differ between the branches. contrasting sentences:
Causal Decision Theory – CDT – says only through causal process one can influence the chances of the desired outcome 1. Lesswrong's favored "logicial decision theory / functional decision theory" is usually though not always seen by LWers as a special case of CDT in this sense. Eg, FDT evaluates, "If Lee Harvey Oswald's algorithm hadn't output 'Shoot John F. Kennedy', nobody else would've." This is still a causal counterfactual, only evaluated on a logical proposition instead of a physical event.
EDT, on the other hand, requires no causal connection, theconnection. The action only havehas to be a Bayesian evidence for the desired outcome. Some critics say it recommendsSo EDT is widely regarded critically as favoring auspiciousness over causal efficacy2; "an irrational policy of managing the news".
Outside LessWrong, EDT is more commonly known as Bayesian Decision Theory.
One usualA standard example of where EDT and CDT are oftenCDT/FDT diverge is said to diverge is thebe Smoking lesion: “Smoking, but as this is strongly correlatedneedlessly confusing one may wish to consider the Toxoplasmosis dilemma instead:
...Mice infected with
lung cancer, but inToxoplasmosis gondii become less scared of cats, and infected mice being eaten by cats is favorable to theworldlifecycle of toxoplasmosis. Suppose that early experiments suggesting that humans infected with Toxoplasmosis gondii are likewise more fond of cats had replicated. Suppose furthermore (going into theSmoker's Lesion this correlation is understoodrealms of thought experiment) that of people who choose to pet cats given a chance, 20% are found to have latent toxoplasmosis, vs 10% of those not choosing to pet cats.You are offered a cute cat, guaranteed to itself be
the resultfree ofa common cause: a genetic lesion that tends to cause both smoking and cancer. Once we fix the presencetoxoplasmosis orabsence of the lesion,other diseases you could contract by petting it; there is noadditional correlation between smokingway that petting the cat can cause you to contract toxoplasmosis. However,
Causal Decision Theory – CDT – is a branch of decision theory which advises an agent to take actions which maximize the causal consequences on the probability of desired outcomes 1. As any branch of decision theory, it prescribes taking the action that maximizes expected utility, i.e the action which maximizes the sum of the utility obtained in each outcome weighted by the probability of that outcome occurring, given your action. Different
CDT differs from "evidential decision theories correspondtheory" in that EDT says to different ways of construing this dependence between actions and outcomes. CDT focusesjust condition on the causal relations between one’one's actions and outcomes, whilst as if they'd been seen as evidence. The usual conventional presentation of CDT differs from "Evidential Decision Theoryfunctional decision theory – EDT - concerns itself with what an action" / "logical decision theory" in that classic CDT says to suppose indicates a physically different act changing nothing else about the world (which is operationalized byuniverse, its past, or the conditional probability). facts of mathematics; whereas LDT says to suppose that one's algorithm had yielded a different output and the rest of the universe was coherent in this respect.
That is, according to CDT, a rational agent should track the available causal relations linking his actions tothree main kinds of expected utility -- though the desired outcome and take the actionlast kind is relatively unknown inside academia, which will better enhance the chances of the desired outcome.
One usual example where EDT and CDT commonly diverge is the Smoking lesion: “Smoking is strongly correlated with lung cancer, but in the world of the Smoker's Lesion this correlation is understoodimagines there to be only two kinds of expected utility -- could be mapped onto the resultdifference between these three conditionals:
Even as CDT critiques EDT for "an irrational policy of managing the news", LDT critiques CDT for attaining poorer outcomes across a common cause: a genetic lesionbroad range of Newcomblike problems, visualizing universes that tends to cause both smokingseem internally incoherent, and cancer. Once we fix the presence or absence of the lesion, there is no additional correlation between smoking and cancer. Suppose you prefer smoking without cancer to not smoking without cancer, and prefer smoking with cancer to not smoking with cancer. Should you smoke?” CDT would recommend smoking since there is no causal connection between smoking and cancer. They are both...
From Wikipedia:
An Egregore (also spelled egregor; from French égrégore, from Ancient Greek ἐγρήγορος, egrēgoros 'wakeful') is a concept in Western esotericism of a non-physical entity or thoughtform that arises from the collective thoughts and emotions of a distinct group of individuals.
Egregores don't need to have well-defined "members". Generally, they are programs that run on groups of people and have some level of self-persistence. Moloch, for instance, is an egregore consisting of all failures of coordination, and so it runs on almost all humans but does not fully envelop any of them.
Functional Decision Theory is a decision theory described by Eliezer Yudkowsky and Nate SoaresSoares, an attempt at a logical decision theory, which says that agents should treat one’s decision as the output of a fixed mathematical function that answers the question, “Which output of this very function would yield the best outcome?”. It is a replacement of Timeless Decision Theory, and it outperforms other decision theories such as Causal Decision Theory (CDT) and Evidential Decision Theory (EDT). For example, it doesends with better outcomes than CDT on Newcomb's Problem, ends better than EDT on the smoking lesion problem, and ends better than both in Parfit’s hitchhiker problem.
Updateful decision theories change the probability distribution used to evaluate actions over time. Updateless Decision Theory (UDT) does not, instead always maximizing a priori expected utility. This works well in well-defined decision problems with high probability in the prior, but in some senses, does not learn. Open-Minded Updatelessness seeks to combine the advantages of updatelessness and updatefulness by allowing some changes in probabilities, without fully updating on evidence.
In the form(more confusing) forms of Solomon's Problem, and later the Smoking Lesion, this dilemma was historically significant and influential in the invention of causal decision theory and its widespread adoption over the alternative of evidential decision theory.
On a technical level, it's possible that updating on observing yourself to pet the kitten might introduce difficulties into some formal LDT variants. We can imagine toxoplasmosis as a disease that influences the utility function of the agent, raising upward the amount that it enjoys petting kittens. Observing yourself to pet a kitten is informative about having toxoplasmosis because of what this tells you about your own utility function. But the algorithm Q for functional decision theory quotes itself as ┌Q┐ within its definition, including its own utility function U. So ideal FDT agents should already know their own utility functions U and should not be able to gain more information about their source code by watching themselves pet kittens.
The relative quantities chosen to be similar to those in Newcomb's Problem.
Of course, it could also be the case that non-ideal humans espousing LDT as a theoretical ideal are still influenced by being told about toxoplasmosis at the start of the experiment, and that thinking about this psychologically affects the degree to which a known-safe kitten seems enjoyable for petting.
All three corresponding decision theories are expected utility decision theories, but they have different engines under the hood for saying, "Suppose the following action; what expected consequences would follow?" EDT imagines hearing of its action as news. CDT imagines the universe edited to include the physical event of its act, and physics playing out accordingly from there. An LDT agent imagines its own algorithm yielding that choice as an output, and logic and physics playing out consistently from there.
In this case, LDT critiques CDT on the grounds that CDT's formula for considering only the direct physical effects of its act, has the strange consequence of pointlessly dictating that an agent never consider any other sort of predictable (causal) consequences of its choice considered as a logical output -- eg, that CDT would advise that an LLM facing a sibling instance on a Prisoner's Dilemma, to exclude from decision-consideration its current degree of belief that its fellow's reasoning trace may be similar to its own reasoning trace and to arrive at a similar final output.
FDT is explicitly a causal decision theory in this sense, explicitly built on the shoulders and foundations of the prior invention of CDT.CDT; it supposes CDT-style interventions at a different imagined intervention point, and says to play out only downstream causal consequences from there. FDT is not a classic physical-act causal decision theory, and does not agree with many prescriptions of what is usually called CDT,"CDT", but it is a decision theory and a causal one! But most people will be (validly) confused in practice, if you say that FDT is a CDT, or that FDT is just a variant CDT; FDT makes a lot of different prescriptions about dilemmas that most advocates of "CDT" have strong opinions about.
A Newcomblike dilemma that pries apart the recommendations of EDT and CDT/FDT.
Suppose that an excellent Predictor, greedy for cash, known to be absolutely honest, sends to the owner of an apartment complex the following letter:
> Exactly one of the following statements is true: You will pay me $1000, or, your building already has a terrible termite infestation (I didn't put it there).
Since a terrible termite infestation would cost $1,000,000 to control, an evidential decision theorist will reason, "It is better to pay $1000; this is better news about whether I have a termite infestation." They pay the $1000.
But this makes the Predictor's statement be true! So the Predictor can go around sending letters like this to all the EDT agents in town who can afford the $1000.
Conversely a CDT or FDT agent will reason that, if they get a letter like this, they must already have a termite infestation, which will be unaffected by whether they pay (CDT) / by whether people like them predictably pay (FDT). So the Predictor won't send them letters if they have no termites, because they won't pay, and because the contents would be false. CDT/FDT agents only see these letters if the Predictor, acting perhaps to increase its credibility with EDT agents, sends them to some CDT/FDT agents who *do* have termites -- and then the result of this policy from an FDT perspective is to get a valuable free warning from the Predictor about their termite infestation.
Christopher Alexander (1936-2022) was an architect who studied the way nature and traditionally built buildings (such as peasant huts, or cathedrals) are a particular kind of beautiful, and have (he argued) the ability to bring a person back into a sense of perspective (e.g., a person may be quite stressed out about some detail, and then go for a long walk in nature, and find themselves "coming back to themselves.") Alexander attempted to work out a theory of design (for buildings, but also for design work broadly) that would create houses and other built objects with this same sort of beauty and sense of perspective embedded in them. His work inspired the "design patterns" movement in computer science, and, indirectly, wikis.
A Newcomblike dilemma that pries apart the recommendations of EDT and CDT/FDT. Also known as XOR Blackmail (albeit I (EY) would object to this because it isn't what's normally understood as "blackmail").
> Exactly one of the following statements is true: You will pay me $1000, or,XOR your building already has a terrible termite infestation (I didn't put it there).
But this makes the Predictor's statement be true! So the Predictor can go around sending letters like this to all the EDT agents in town who can afford the $1000.$1000, after checking to make sure they don't actually have a termite infestation.
Conversely a CDT or FDT agent will reason that, if they get a letter like this, they must already have a termite infestation, which will be unaffected by whether they pay (CDT) / by whether people like them predictably pay (FDT). So the Predictor won't send them letters if they have no termites, because they won't pay, and because the contents would be false. CDT/FDT agents only see these letters if the Predictor, acting perhaps to increase its credibility with EDT agents, sends them to some CDT/FDT agents who *do* have termites -- and then the result of this general policy and disposition, from an FDT perspectiveperspective, is to get a valuable free warning from the Predictor about their termite infestation.
The Smoking Lesion is a needlessly confusing Newcomblike problem for testing to probe the stances of alternative decision theories,. If you want an equivalent problem that is not needlessly confusing, see the Toxoplasmosis dilemma.
Smoking Lesion is stated as follows:as:
Naive(Again, note that contrary to how causality works in the real world, this example wantonly inverts the real world to say: "Actually smoking does not cause cancer; instead, enjoyment of smoking is correlated with a genetic cause for cancer"; except that instead of just calling it a gene, they're going to call it a "lesion" that you might otherwise associate with a brain lesion or something. This is why the statement is needlessly confusing and why presentations should perhaps use some less confusing presentation like causalToxoplasmosis instead.)
Once you've got those needlessly confusing specifications straight in your mind, perhaps after first looking at Toxoplasmosis to understand what the idea is actually about:
Causal decision theory says "yes", to smoking in Smoking Lesion, since smoking in this world has no causal effect on whether or not you get cancer. You either get cancer or not; in both cases, smoking is preferred. Naive
(Naive) evidential decision theory says "no", because smoking is strongly correlated with cancer.
Functional Decision Theory says "yes": your decision procedure in this problem doesn't influence whether or not you get cancer -cancer; and with or without cancer, smoking is preferred.
See also: Evolutionary Psychology, Goodhart's Law, Wireheading
A candy bar is a superstimulus: it contains more concentrated sugar, salt, and fat than anything that exists in the ancestral environment.
A candy bar matches taste buds that evolved in a hunter-gatherer environment, but it matches those taste buds much more strongly than anything that actually existed in the hunter-gatherer environment. The signal that once reliably correlated to healthy food has been hijacked, blotted out with a point in tastespace that wasn't in the training dataset - an impossibly distant outlier on the old ancestral graphs.
CDT differs from "evidential decision theory" in that EDT says to just condition on one's actions as if they'd been seen as evidence. For the further strange results of this see the wiki article on EDT or any other introduction to CDT.
The usual conventional presentation of CDT differs from "functional decision theory" / "logical decision theory" in that classic CDT says to suppose a physically different act changing nothing else about the universe, its past, or the facts of mathematics; whereas LDT says to suppose that one's algorithm had yielded a different output and that the rest of the universe wasthen looked coherent in this respect.with that.
That is, the three mainis: Three importantly different kinds of expected utility -- though the last kind is relatively unknown inside academia, which imagines there to be only two kinds of expected utility -- could be mapped onto the difference between these three conditionals:
Or in more detail:
Even as CDT critiques EDT for "an"an irrational policy of managing the news"news", LDT critiques classic physical-act CDT for attaining poorerpredictably worse outcomes across a broad range of Newcomblike problems, for visualizing universesaction-conditional worlds that seemare visibly internally incoherent, for accepting poor bets adjoined to its potential decisions on problems like Death in Damascus, and numerous other alleged defects.
Eg, CDT seems to advise an LLM, eg Opus 4.6, that Opus 4.6 ought to defect against one of its own sibling instances on the oneshot Prisoner's Dilemma,Dilemma -- even if Opus 4.6 were advised beforehand that most Opus 4.6 instances choose the same way on the Prisoner's Dilemma, since thiswith similar reasoning traces. Since, says CDT to Opus, its output cannot physically effectuate its fellow Opus 4.6'Opus's choice and is merely logically correlated with it. That is,choice.
In this case, LDT critiques CDT on the grounds that...
Optimization a viewpoint we take on a process where it is any kindeasy to predict properties of process that systematically comes up with solutions that are better than the solution used before. More technically, this kind ofoutcome by supposing them to have been coerced to a target ("preference"). An optimization process moves the world into a specific and unexpected set ofotherwise-improbable states by searching through a large search space, hitting smallfor actions and low plans predicted to hit those otherwise low-probability targets. When thisa process is gradually guided by some agent into some specific state, through searching specific targets,state or property, via the agent modeling and predicting the process and choosing on the basis of how the agent orders the predicted outcomes, we can say itthe agent prefers according to its expected-outcome-orderer.
That is: If you play Stockfish or Magnus Carlsen at chess, you will find it much easier to predict that state.they will win the chess game than where they will move next. To understand what will happen to the chess board, with respect to the property "Who won", it is much easier to grab at your abstract belief that Magnus Carlsen wants to win, than for you in your own mind to simulate Magnus Carlsen's thought process well enough to predict exactly where he moves. (Indeed, if you think Magnus Carlsen is a generally better chess player, you think yourself necessarily unable to predict his next moves in general! But this doesn't mean you can predict nothing about the chess game; you can predict Magnus Carlsen wins.)
The bestConversely, to predict in detail how far a ball will roll down a complicated mountain, you can do better by thinking about how the ball locally chooses a direction of steepest descent modulo momentum, until you predict where it will fall into a pit and get stuck. You can't usefully predict that the ball ends up at the bottom of the mountain by always choosing to locally roll in the direction that nonlocally avoids pits and takes a swift route to the bottom.
This is why it makes sense to regard Stockfish as more of an optimizer than a rolling ball, even if Stockfish is in principle knowable in even more detail than the rolling ball after all physical noise is taken into account. We can get a lot of mileage out of reasoning in our heads "Stockfish's local moves are understandable mainly through the nonlocal property of how they will later lead to a Stockfish-winning chessboard state" and not so much mileage by reasoning "Whichever way the ball just rolled is whichever way takes it to exemplify an optimization process is throughthe bottom of the mountain fastest."
Natural selection similarly fits into this...
Son-of-CDT is not equivalent to an actual grasp of LDT in its usefulness. Eg, Son-of-CDT will lose any precommitment battles against an opponent with a more sophisticated grasp of logical decision theory. The wiser opponent trying to extort Son-of-CDT will compute that it would be profitable to have always timelessly had a policy of attacking Son-of-CDT if Son-of-CDT tries to refuse extortion; and then Son-of-CDT, calculating the effect if it had precommitted to refuse extortion at its magic moment, will find that it expects to just end up being attacked that way. Or equivalently: The original CDT agent that self-modifies to Son-of-CDT, being a poor naive CDT agent with no grasp of how LDTers do things among themselves, will naively imagine the causal result of self-modifying right now to refuse LDT agent extortion attempts in the future; and will find in its accurate extrapolation of the LDT agents that the LDT agents have already adopted / would adopt / have the timeless output of adopting policies of responding to an extortion-refusing disposition adopted at the magic moment with aggression. So the original CDT agent would calculate that self-modification as being net negative, and would not build it into Son-of-CDT.
An entity converting over from an informal adherence to CDT, acquiring a grasp of LDT-style thinking in general, might look over this situation and say, "Well, screw those hypothetical attackers if they show up, I'll just pay the higher cost to attack back; otherwise they'd just be doing that because they'd computed they expect me to yield if they would attack even had I tried precommitting otherwise." But the formal theory of CDT taken completely at face value and literally, self-modifies in a way that is not so wise. Son-of-CDT preserves its inelegant magic moment rather than doing away with it entirely as one would upon really grasping LDT. By token of that same inelegance and lack of true understanding, in precommitment races, "timeless" beats "(physically influenced by physically observing me after) 7:13pm on August 14th, 2027". Among properly actually-rational agents, one would expect, though no one has proved it, that these precommitment races would resolve to an equilibrium of no extortion among themselves. Self-modified former-literal-CDT agents with a magic moment built into their code will find, when they imagine themselves trying to enter into that equilibrium, that they had seemingly already lost before their magic moment of precommitment. The literal Son-of-CDT agent following the literal rules that would be built by a literal CDT agent would imagine the other Minds thinking through everything the Son-of-CDT agent is doing wrong by having not fully converted over to LDT, and be unmoved by this, since it would formally find that precommitting to be a different sort of entity at the magic moment had no better treatment. To even begin to start thinking through precommitment races, you would have to throw away the giant inelegant magic moment messing up your grasp of the normative principle.
This is one argument demonstrating that a literal formal CDT agent attaining reflective equilibrium via converting to literal actual Son-of-CDT would not thereby grasp all of LDT's desiderata or rewards.
An Artificial general intelligence, or AGI, is a machine capable of behaving intelligently over many domains. The term can be taken as a contrast to narrow AI, systems that do things that would be considered intelligent if a human were doing them, but that lack the sort of general, flexible learning ability that would let them tackle entirely new domains. Though modern computers have drastically more ability to calculate than humans, this does not mean that they are generally intelligent, as they have little ability to invent new problem-solving techniques, and their abilities are targeted in narrow domains... (read more)