Superintelligence via whole brain emulation

8 AlexMennen 17 August 2016 04:11AM

Most planning around AI risk seems to start from the premise that superintelligence will come from de novo AGI before whole brain emulation becomes possible. I haven't seen any analysis that assumes both uploads-first and the AI FOOM thesis (Edit: apparently I fail at literature searching), a deficiency that I'll try to get a start on correcting in this post.

It is likely possible to use evolutionary algorithms to efficiently modify uploaded brains. If so, uploads would likely be able to set off an intelligence explosion by running evolutionary algorithms on themselves, selecting for something like higher general intelligence.

Since brains are poorly understood, it would likely be very difficult to select for higher intelligence without causing significant value drift. Thus, setting off an intelligence explosion in that way would probably produce unfriendly AI if done carelessly. On the other hand, at some point, the modified upload would reach a point where it is capable of figuring out how to improve itself without causing a significant amount of further value drift, and it may be possible to reach that point before too much value drift had already taken place. The expected amount of value drift can be decreased by having long generations between iterations of the evolutionary algorithm, to give the improved brains more time to figure out how to modify the evolutionary algorithm to minimize further value drift.

Another possibility is that such an evolutionary algorithm could be used to create brains that are smarter than humans but not by very much, and hopefully with values not too divergent from ours, who would then stop using the evolutionary algorithm and start using their intellects to research de novo Friendly AI, if that ends up looking easier than continuing to run the evolutionary algorithm without too much further value drift.

The strategies of using slow iterations of the evolutionary algorithm, or stopping it after not too long, require coordination among everyone capable of making such modifications to uploads. Thus, it seems safer for whole brain emulation technology to be either heavily regulated or owned by a monopoly, rather than being widely available and unregulated. This closely parallels the AI openness debate, and I'd expect people more concerned with bad actors relative to accidents to disagree.

With de novo artificial superintelligence, the overwhelmingly most likely outcomes are the optimal achievable outcome (if we manage to align its goals with ours) and extinction (if we don't). But uploads start out with human values, and when creating a superintelligence by modifying uploads, the goal would be to not corrupt them too much in the process. Since its values could get partially corrupted, an intelligence explosion that starts with an upload seems much more likely to result in outcomes that are both significantly worse than optimal and significantly better than extinction. Since human brains also already have a capacity for malice, this process also seems slightly more likely to result in outcomes worse than extinction.

The early ways to upload brains will probably be destructive, and may be very risky. Thus the first uploads may be selected for high risk-tolerance. Running an evolutionary algorithm on an uploaded brain would probably involve creating a large number of psychologically broken copies, since the average change to a brain will be negative. Thus the uploads that run evolutionary algorithms on themselves will be selected for not being horrified by this. Both of these selection effects seem like they would select against people who would take caution and goal stability seriously (uploads that run evolutionary algorithms on themselves would also be selected for being okay with creating and deleting spur copies, but this doesn't obviously correlate in either direction with caution). This could be partially mitigated by a monopoly on brain emulation technology. A possible (but probably smaller) source of positive selection is that currently, people who are enthusiastic about uploading their brains correlate strongly with people who are concerned about AI safety, and this correlation may continue once whole brain emulation technology is actually available.

Assuming that hardware speed is not close to being a limiting factor for whole brain emulation, emulations will be able to run at much faster than human speed. This should make emulations better able to monitor the behavior of AIs. Unless we develop ways of evaluating the capabilities of human brains that are much faster than giving them time to attempt difficult tasks, running evolutionary algorithms on brain emulations could only be done very slowly in subjective time (even though it may be quite fast in objective time), which would give emulations a significant advantage in monitoring such a process.

Although there are effects going in both directions, it seems like the uploads-first scenario is probably safer than de novo AI. If this is the case, then it might make sense to accelerate technologies that are needed for whole brain emulation if there are tractable ways of doing so. On the other hand, it is possible that technologies that are useful for whole brain emulation would also be useful for neuromorphic AI, which is probably very unsafe, since it is not amenable to formal verification or being given explicit goals (and unlike emulations, they don't start off already having human goals). Thus, it is probably important to be careful about not accelerating non-WBE neuromorphic AI while attempting to accelerate whole brain emulation. For instance, it seems plausible to me that getting better models of neurons would be useful for creating neuromorphic AIs while better brain scanning would not, and both technologies are necessary for brain uploading, so if that is true, it may make sense to work on improving brain scanning but not on improving neural models.

Two kinds of population ethics, and Current-Population Utilitarianism

7 AlexMennen 17 June 2014 10:26PM

There are two different kinds of questions that could be considered to fall under the subject of population ethics: “What sorts of altruistic preferences do I have about the well-being of others?”, and “Given all the preferences of each individual, how should we compromise?”. In other words, the first question asks how everyone's experiential utility functions (which measure quality of life) contribute to my (or your) decision-theoretic utility function (which takes into account everything that I or you, respectively, care about), and the second asks how we should agree to aggregate our decision-theoretic utility functions into something that we can jointly optimize for. When people talk about population ethics, they often do not make it clear which of these they are referring to, but they are different questions, and I think the difference is important.

 

For example, suppose Alice, Bob, and Charlie are collaborating on a project to create an artificial superintelligence that will take over the universe and optimize it according to their preferences. But they face a problem: they have different preferences. Alice is a total utilitarian, so she wants to maximize the sum of everyone's experiential utility. Bob is an average utilitarian, so he wants to maximize the average of everyone's experiential utility. Charlie is an egoist, so he wants to maximize his own experiential utility. As a result, Alice, Bob, and Charlie have some disagreements over how their AI should handle decisions that affect the number of people in existence, or which involve tradeoffs between Charlie and people other than Charlie. They at first try to convince each other of the correctness of their view, but they eventually realize that they don't actually have any factual disagreement; they just value different things. As a compromise, They program their AI to maximize the average of everyone's experiential utility, plus half of Charlie's experiential utility, plus a trillionth of the sum of everyone's experiential utility.

 

Of course, there are other ways for utility functions to differ than average versus total utilitarianism and altruism versus egoism. Maybe you care about something other than the experiences of yourself and others. Or maybe your altruistic preferences about someone else's experiences differs from their selfish preferences, like how a crack addict wants to get more crack while their family wants them not to.

 

Anyway, the point is, there are many ways to aggregate everyone's experiential utility functions, and not everyone will agree on one of them. In fact, since people can care about things other than experiences, many people might not like any of them. It seems silly to suggest that we would want a Friendly AI to maximize an aggregation of everyone's experiential utility functions; there would be potentially irresolvable disagreements over which aggregation to use, and any of them would exclude non-experiential preferences. Since decision-theoretic utility functions actually take into account all of an agent's preferences, it makes much more sense to try to get a superintelligence to maximize an aggregation of decision-theoretic utility functions.

 

The obvious next question is which aggregation of decision-theoretic utility functions to use. One might think that average and total utilitarianism could both be applied to decision-theoretic utility functions, but that is actually not so easy. Decision-theoretic utility functions take into account everything the agent cares about, which can include things that happen in the far future, after the agent dies. With a dynamic population, it is unclear which utility functions should be included in the aggregation. Should every agent that does or ever will exist have their utility function included? If so, then the aggregation would indicate that humans should be replaced with large numbers of agents whose preferences are easier to satisfy1 (this is true even for average utilitarianism, because there needs to be enough of these agents to drown out the difficult-to-satisfy human preferences in the aggregation). Should the aggregation be dynamic with the population, so that at time t, the preferences of agents who exist at time t are taken into account? That would be dynamically inconsistent. In a population of sadists who want to torture people (but only people who don't want to be tortured), the aggregation would indicate that they should create some people and then torture them. But then once the new people are created, the aggregation would take their preferences into account and indicate that they should not be tortured.

 

I suggest a variant that I'm tentatively calling current-population utilitarianism: Aggregate the preferences of the people who are alive right now, and then leave this aggregated utility function fixed even as the population and their preferences change. By “right now”, I don't mean June 17, 2014 at 10:26 pm GMT; I mean the late pre-singularity era as a whole. Why? Because this is when the people who have the power to affect the creation of the AGI that we will want to maximize said aggregated utility function live. If it were just up to me, I would program an AGI to maximize my own utility function2, but one person cannot do that on their own, and I don't expect I'd be able to get very many other people to go along with that. But all the people who will be contributing to an FAI project, and everyone whose support they can seek, all live in the near-present. No one else can support or undermine an FAI project, so why make any sacrifices for them for any reason other than that you (or someone who can support or undermine you) care about them (in which case their preferences will show up in the aggregation through your utility function)? Now I'll address some anticipated objections.

 

Objection: Doesn't that mean that people created post-singularity will be discriminated against?

Answer: To the extent that you want people created post-singularity not to be discriminated against, this will be included in your utility function.

 

Objection: What about social progress? Cultural values change over time, and only taking into account the preferences of people alive now would force cultural values to stagnate.

Answer: To the extent that you want cultural values to be able to drift, this will be included in your utility function.

 

Objection: What if my utility function changes in the future?

Answer: To the extent that you want your future utility function to be satisfied, this will be included in your utility function.

 

Objection: Poor third-worlders also cannot support or undermine an FAI project. Why include them but not people created post-singularity?

Answer: Getting public support requires some degree of political correctness. If we tried to rally people around the cause of creating a superintelligence that will maximize the preferences of rich first-worlders, I don't think that would go over very well.

 

One utility function being easier to satisfy than another doesn't actually mean anything without some way of normalizing the utility function, but since aggregations require somehow normalizing the utility functions anyway, I'll ignore that problem.

This is not a proclamation of extreme selfishness. I'm still talking about my decision-theoretic utility function, which is defined, roughly speaking, as what I would maximize if I had godlike powers, and is at least somewhat altruistic.

Selfish reasons to reject the repugnant conclusion in practice

4 AlexMennen 05 June 2014 06:41AM

Prerequisite: http://en.wikipedia.org/wiki/Mere_addition_paradox

 

The repugnant conclusion is the proposition that for any well-off population A, and for any quality of life Q which exceeds the minimal quality of a life worth creating (no matter how small the margin), there exists a possible population B with quality of life Q such that it is better from a utilitarian standpoint for B to exist than it is for A to exist. I find the repugnant conclusion convincing, for reasons which have been discussed elsewhere and I will not repeat here.

 

Consider the path from population A to population A+ to population B described in the linked wikipedia article from the perspectives of members of the above populations. To a member of population A who values the creation of lives worth living, the transition from A to A+ by creating lives worth living is an improvement. From the perspective of an average member of population A+, the transition from A+ to B is neutral; you might gain and you might lose, but on average you break even, and from an altruistic standpoint, it's also break-even. But from the perspective of a member of population A, the composite transition all the way from A to B doesn't look so great. If you accept the repugnant conclusion, then it's a win from an altruistic perspective, but you personally lose out. It may or may not still seem like a good idea overall. (Similarly, even if you personally are not affected, people also care about their friends and family more than about arbitrary strangers, and these people that actually-existing people especially care about are disproportionately likely to also actually exist, so this could be another reason not to transition from A to B, but for simplicity, I'll ignore such considerations for the rest of this post.)

 

Here we have a variant of the repugnant conclusion: the proposition that for any well-off population A, and for any quality of life Q which exceeds the minimal quality of a life worth creating (no matter how small the margin), there exists a possible population B with quality of life Q such that, if you are a member of A, you should, if given the chance, replace population A with population B and become an arbitrary member of B. Since decisions that affect populations tend to be made by people who already exist at the time, it makes sense to frame population ethics questions from the perspective of members of the population like this.

 

Let P be a variable measuring your personal well-being, T be the sum of variables measuring the the well-beings of everyone in the population, c be a positive constant, and f be a bounded, monotonically increasing function. Most people are at least somewhat altruistic, but also care about themselves much more than about others, so let's assume that your utility function increases monotonically with P and T. Consider the possible utility functions  and . For scenarios in which the total population is fixed, both of these utility functions can be made arbitrarily selfish or altruistic by adjusting f and c, and for scenarios in which your choices only affect T (P is constant) and all options are deterministic, both utility functions will say to act like total utilitarians. In particular, both of them accept the repugnant conclusion. But while  accepts the variant,  does not. That is, if you accept the abstract repugnant conclusion, whether or not you would want to implement it in practice depends on not only how selfish or altruistic you are, but also the manner in which you are selfish and altruistic. I don't like the idea of caring about myself arbitrarily little as the population grows to infinity, so it seems intuitive to me that my utility function would be more like  than like .

Prisoner's dilemma tournament results

32 AlexMennen 09 July 2013 08:50PM

The prisoner's dilemma tournament is over. There were a total of 21 entries. The winner is Margaret Sy, with a total of 39 points. 2nd and 3rd place go to rpglover64 and THE BLACK KNIGHT, with scores of 38 and 36 points respectively. There were some fairly intricate strategies in the tournament, but all three of these top scorers submitted programs that completely ignored the source code of the other player and acted randomly, with the winner having a bias towards defecting.

You can download a chart describing the outcomes here, and the source codes for the entries can be downloaded here.

I represented each submission with a single letter while running the tournament. Here is a directory of the entries, along with their scores: (some people gave me a term to refer to the player by, while others gave me a term to refer to the program. I went with whatever they gave me, and if they gave me both, I put the player first and then the program)

A: rpglover64 (38)
B: Watson Ladd (27)
c: THE BLACK KNIGHT (36)
D: skepsci (24)
E: Devin Bayer (30)
F: Billy, Mimic-- (27)
G: itaibn (34)
H: CooperateBot (24)
I: Sean Nolan (28)
J: oaz (26)
K: selbram (34)
L: Alexei (25)
M: LEmma (25)
N: BloodyShrimp (34)
O: caa (32)
P: nshepperd (25)
Q: Margaret Sy (39)
R: So8res, NateBot (33)
S: Quinn (33)
T: HonoreDB (23)
U: SlappedTogetherAtTheLastMinuteBot (20)


Prisoner's Dilemma (with visible source code) Tournament

47 AlexMennen 07 June 2013 08:30AM

After the iterated prisoner's dilemma tournament organized by prase two years ago, there was discussion of running tournaments for several variants, including one in which two players submit programs, each of which are given the source code of the other player's program, and outputs either “cooperate” or “defect”. However, as far as I know, no such tournament has been run until now.

Here's how it's going to work: Each player will submit a file containing a single Scheme lambda-function. The function should take one input. Your program will play exactly one round against each other program submitted (not including itself). In each round, two programs will be run, each given the source code of the other as input, and will be expected to return either of the symbols “C” or “D” (for "cooperate" and "defect", respectively). The programs will receive points based on the following payoff matrix:

“Other” includes any result other than returning “C” or “D”, including failing to terminate, throwing an exception, and even returning the string “Cooperate”. Notice that “Other” results in a worst-of-both-worlds scenario where you get the same payoff as you would have if you cooperated, but the other player gets the same payoff as if you had defected. This is an attempt to ensure that no one ever has incentive for their program to fail to run properly, or to trick another program into doing so.

Your score is the sum of the number of points you earn in each round. The player with the highest score wins the tournament. Edit: There is a 0.5 bitcoin prize being offered for the winner. Thanks, VincentYu!

Details:
All submissions must be emailed to wardenPD@gmail.com by July 5, at noon PDT (Edit: that's 19:00 UTC). Your email should also say how you would like to be identified when I announce the tournament results.
Each program will be allowed to run for 10 seconds. If it has not returned either “C” or “D” by then, it will be stopped, and treated as returning “Other”. For consistency, I will have Scheme collect garbage right before each run.
One submission per person or team. No person may contribute to more than one entry. Edit: This also means no copying from each others' source code. Describing the behavior of your program to others is okay.
I will be running the submissions in Racket. You may be interested in how Racket handles time (especially the (current-milliseconds) function), threads (in particular, “thread”, “kill-thread”, “sleep”, and “thread-dead?”), and possibly randomness.
Don't try to open the file you wrote your program in (or any other file, for that matter). I'll add code to the file before running it, so if you want your program to use a copy of your source code, you will need to use a quine. Edit: No I/O of any sort.
Unless you tell me otherwise, I assume I have permission to publish your code after the contest.
You are encouraged to discuss strategies for achieving mutual cooperation in the comments thread.
I'm hoping to get as many entries as possible. If you know someone who might be interested in this, please tell them.
It's possible that I've said something stupid that I'll have to change or clarify, so you might want to come back to this page again occasionally to look for changes to the rules. Any edits will be bolded, and I'll try not to change anything too drastically, or make any edits late in the contest.

Here is an example of a correct entry, which cooperates with you if and only if you would cooperate with a program that always cooperates (actually, if and only if you would cooperate with one particular program that always cooperates):

(lambda (x)
    (if (eq? ((eval x) '(lambda (y) 'C)) 'C)
        'C
        'D))

VNM agents and lotteries involving an infinite number of possible outcomes

9 AlexMennen 21 February 2013 09:58PM

Summary: The VNM utility theorem only applies to lotteries that involve a finite number of possible outcomes. If an agent maximizes the expected value of a utility function when considering lotteries that involve a potentially infinite number of outcomes as well, then its utility function must be bounded.

Outcomes versus Lotteries

One way to formulate the VNM utility theorem is in terms of outcomes and lotteries over outcomes. That is, there is some set of outcomes, and a set of lotteries defined as . In other words, the set of lotteries is the set of probability distributions over a finite number of outcomes. The finiteness is very important; we'll get to that later. Note that for each outcome, there is a corresponding lottery that guarantees this outcome, and these “pure outcome” lotteries are a basis for .

Given that formulation, and given the VNM axioms, there exists some function  such that given any 2 lotteries  and iff .

The other formulation does not mention . Instead, there is simply a set of lotteries, such that  iff . In this formulation, there exists some function  such that if , then  (notice  still must be finite) and for any 2 lotteries  and , iff .

The formulation in terms of outcomes and lotteries over outcomes is more intuitively appealing (to me, at least), since real life has outcomes and uncertainty about outcomes, so I will use it when I can, but the formulation purely in terms of lotteries, which is more similar to what von Neumann and Morgenstern did in their original paper, will be useful sometimes, so I will switch back to it intermittently.

Infinite lotteries

Myth: Given some utility function that accurately describes a VNM-rational agent's preferences over finite lotteries, if you expand to include lotteries with an infinite number of possible outcomes (let's call the expanded set of lotteries ), then for any 2 lotteries  and , iff .

Reality: Knowing an agent's preferences over finite lotteries, and that the agent obeys the VNM axioms, does not tell you everything about the agent's preferences over lotteries with an infinite number of possible outcomes. To demonstrate this, I'm going to construct a VNM-rational agent that maximizes a utility function , where . This construction relies on the axiom of choice (please let me know if you figure out whether or not it is possible to construct such an agent without the axiom of choice). I will also be assuming that is countably infinite (if is finite, such an agent is impossible, and if it is uncountable, then you can consider a countable subset).

Notice that can be seen as a subset of the real vector space , with the addition and multiplication by scalar operations being exactly what you would expect (, and ). A utility function  can be seen as an element of the dual space of . The axiom of choice implies that this vector space has a basis (in this context, a basis means a set of vectors for which any finite subset is linearly independent, and every vector is a linear combination of a finite number of basis vectors). The value of on each basis element can be chosen independently, and these choices completely determine . In particular, the basis could contain every element of , and also contain  for some sequence  with distinct elements. Then we could have  and , violating the conclusion of the myth, but this meets all the VNM axioms.

The fact that there is a real-valued function on lotteries that the agent maximizes guarantees that the completeness and transitivity axioms hold, since  or  or  (completeness), and if  and  then  (transitivity). The fact that the function is linear with respect to finite sums guarantees that the continuity and independence axioms hold, since if  then  (continuity), and if then for any lottery  and positive probability (independence).

Extended VNM hypothesis

The VNM utility theorem does not prove that an agent meeting its axioms will maximize the expected value of a utility function when presented with infinite lotteries, but the fact that any such agent will maximize the expected value of a utility function when presented with finite lotteries certainly seems very suggestive. With that in mind, I suggest that this be called the “extended von Neumann-Morgenstern hypothesis”:

An agent, in order to be considered rational, should maximize the expected value of a utility function over outcomes when choosing between lotteries over any (possibly infinite) number of outcomes.

Bounded and unbounded utility functions

It is perfectly possible to construct VNM-rational agents with an unbounded utility function. But all such agents will inevitably violate the extended VNM hypothesis, because it is possible to create infinite lotteries with undefined expected value. For instance, the St. Petersburg paradox can be modified to refer specifically to utilities instead of money. That is, if there is no upper bound to the agent's utility function, then there exists a sequence of outcomes  such that for each , . Then the expected utility of  is , which does not converge. So unbounded utility functions are not compatible with the extended VNM hypothesis.

At this point, one may feel a strong temptation to come up with some way to characterize the values of infinite sums with some ordered superset of the real numbers, so that it is possible to compare nonconvergent sums. However, by the formulation of the VNM theorem solely in terms of lotteries, the utility of any lottery, such as , is a real number. So any such scheme that requires that the range of the utility function include nonreals will violate the VNM axioms. In particular, it will probably violate the archimedian axiom.

One possible response to this is to dismiss the archimedian axiom, and try to characterize agents that obey the completeness, transitivity, and independence axioms. Benja has written (section "Doing without Continuity) about this, and I find his solution fairly compelling, but it isn't clear that it helps us deal with situations like the St. Petersburg paradox. I intend to say more about nonarchimedian preferences soon.

Harsanyi's Social Aggregation Theorem and what it means for CEV

21 AlexMennen 05 January 2013 09:38PM

A Friendly AI would have to be able to aggregate each person's preferences into one utility function. The most straightforward and obvious way to do this is to agree on some way to normalize each individual's utility function, and then add them up. But many people don't like this, usually for reasons involving utility monsters. If you are one of these people, then you better learn to like it, because according to Harsanyi's Social Aggregation Theorem, any alternative can result in the supposedly Friendly AI making a choice that is bad for every member of the population. More formally,

Axiom 1: Every person, and the FAI, are VNM-rational agents.

Axiom 2: Given any two choices A and B such that every person prefers A over B, then the FAI prefers A over B.

Axiom 3: There exist two choices A and B such that every person prefers A over B.

(Edit: Note that I'm assuming a fixed population with fixed preferences. This still seems reasonable, because we wouldn't want the FAI to be dynamically inconsistent, so it would have to draw its values from a fixed population, such as the people alive now. Alternatively, even if you want the FAI to aggregate the preferences of a changing population, the theorem still applies, but this comes with it's own problems, such as giving people (possibly including the FAI) incentives to create, destroy, and modify other people to make the aggregated utility function more favorable to them.)

Give each person a unique integer label from to , where is the number of people. For each person , let be some function that, interpreted as a utility function, accurately describes 's preferences (there exists such a function by the VNM utility theorem). Note that I want to be some particular function, distinct from, for instance, , even though and represent the same utility function. This is so it makes sense to add them.

Theorem: The FAI maximizes the expected value of , for some set of scalars .

Actually, I changed the axioms a little bit. Harsanyi originally used “Given any two choices A and B such that every person is indifferent between A and B, the FAI is indifferent between A and B” in place of my axioms 2 and 3 (also he didn't call it an FAI, of course). For the proof (from Harsanyi's axioms), see section III of Harsanyi (1955), or section 2 of Hammond (1992). Hammond claims that his proof is simpler, but he uses jargon that scared me, and I found Harsanyi's proof to be fairly straightforward.

Harsanyi's axioms seem fairly reasonable to me, but I can imagine someone objecting, “But if no one else cares, what's wrong with the FAI having a preference anyway. It's not like that would harm us.” I will concede that there is no harm in allowing the FAI to have a weak preference one way or another, but if the FAI has a strong preference, that being the only thing that is reflected in the utility function, and if axiom 3 is true, then axiom 2 is violated.

proof that my axioms imply Harsanyi's: Let A and B be any two choices such that every person is indifferent between A and B. By axiom 3, there exists choices C and D such that every person prefers C over D. Now consider the lotteries and , for . Notice that every person prefers the first lottery to the second, so by axiom 2, the FAI prefers the first lottery. This remains true for arbitrarily small , so by continuity, the FAI must not prefer the second lottery for ; that is, the FAI must not prefer B over A. We can “sweeten the pot” in favor of B the same way, so by the same reasoning, the FAI must not prefer A over B.

So why should you accept my axioms?

Axiom 1: The VNM utility axioms are widely agreed to be necessary for any rational agent.

Axiom 2: There's something a little rediculous about claiming that every member of a group prefers A to B, but that the group in aggregate does not prefer A to B.

Axiom 3: This axiom is just to establish that it is even possible to aggregate the utility functions in a way that violates axiom 2. So essentially, the theorem is “If it is possible for anything to go horribly wrong, and the FAI does not maximize a linear combination of the people's utility functions, then something will go horribly wrong.” Also, axiom 3 will almost always be true, because it is true when the utility functions are linearly independent, and almost all finite sets of functions are linearly independent. There are terrorists who hate your freedom, but even they care at least a little bit about something other than the opposite of what you care about.

At this point, you might be protesting, “But what about equality? That's definitely a good thing, right? I want something in the FAI's utility function that accounts for equality.” Equality is a good thing, but only because we are risk averse, and risk aversion is already accounted for in the individual utility functions. People often talk about equality being valuable even after accounting for risk aversion, but as Harsanyi's theorem shows, if you do add an extra term in the FAI's utility function to account for equality, then you risk designing an FAI that makes a choice that humanity unanimously disagrees with. Is this extra equality term so important to you that you would be willing to accept that?

Remember that VNM utility has a precise decision-theoretic meaning. Twice as much utility does not correspond to your intuitions about what “twice as much goodness” means. Your intuitions about the best way to distribute goodness to people will not necessarily be good ways to distribute utility. The axioms I used were extremely rudimentary, whereas the intuition that generated "there should be a term for equality or something" is untrustworthy. If they come into conflict, you can't keep all of them. I don't see any way to justify giving up axioms 1 or 2, and axiom 3 will likely remain true whether you want it to or not, so you should probably give up whatever else you wanted to add to the FAI's utility function.

Citations:

Harsanyi, John C. "Cardinal welfare, individualistic ethics, and interpersonal comparisons of utility." The Journal of Political Economy (1955): 309-321.

Hammond, Peter J. "Harsanyi’s utilitarian theorem: A simpler proof and some ethical connotations." IN R. SELTEN (ED.) RATIONAL INTERACTION: ESSAYS IN HONOR OF JOHN HARSANYI. 1992.

Interpersonal and intrapersonal utility comparisons

11 AlexMennen 04 January 2013 03:05AM

Utility functions are only defined up to an additive constant and a positive multiplier. For example, if we have a simple universe with only 3 possible states (X, Y, and Z), a utility function u such that u(X)=0, u(Y)=1, and u(Z)=3, and another utility function w such that w(X)=-1, w(Y)=1, and w(Z)=5, then as utility functions, u and w are identical, since w=2u-1.

Preference utilitarianism suggests maximizing the sum of everyone's utility function. But the fact that utility functions are invariant on multiplication by positive scalars makes this operation poorly defined. For example, suppose your utility function is u (as defined above), and the only other morally relevant agent has a utility function v such that v(X)=0, v(Y)=2000, and v(Z)=1000. He argues that according to utilitarianism, Y is the best state of the universe, since if you add each of your utility functions, you get (u+v)(X)=0, (u+v)(Y)=2001, and (u+v)(Z)=1003. You complain that he cheated by multiplying his utility function by a large number, and that if you treat v as v(X)=0, v(Y)=2, and v(Z)=1, then Z is the best state of the universe according to utilitarianism. There is no objective way to resolve this dispute, but anyone who wants to build a preference utilitarianism machine has to find a way to resolve such disputes that gives reasonable results.

I'm pretty sure that the idea of the previous two paragraphs has been talked about before, but I can't find where. [Edit: here and here]

Anyway, one might argue that if you are not a preference utilitarian, and not planning to build a friendly AI, you have little reason to care about this problem. If you just want to maximize your personal utility function, surely you don't need a solution to that problem, right?

Wrong! Unless you know exactly what your preferences are, which humans don't. If you're unsure whether or not u or v (as described above) describes your true preferences, and you assign a 50% probability to each, then you face the same problem that preference utilitarianism did in the previous example.

Humans are a lot better at getting ordinal utilities straight than they are at figuring out cardinal utilities, but even assuming that you know the order of your preferences, the problems remain. Let's say that, in another 3-state world (with states A, B, and C) you know you prefer B over A, and C over B, but you are uncertain between the possibilities that you prefer C over A by twice the margin that you prefer B over A, and that you prefer C over A by 10 times the margin that you prefer B over A. You assign a 50% probability to each. Now suppose you face a choice between B and a lottery that has a 20% chance of giving you C and an 80% chance of giving you A. If you define the utility of A as 0 utils and the utility of B as 1 util, then the utility values (in utils) are u1(A)=0, u1(B)=1, u1(C)=2, u2(A)=0, u2(B)=1, u2(C)=10, so the expected utility of choosing B is 1 util, and the expected utility of the lottery is .5*(.2*2 + .8*0) + .5*(.2*10 + .8*0) = 1.2 utils, so the lottery is better. But if you instead define the utility of A as 0 utils and the utility of C as 1 util, then u1(A)=0, u1(B)=.5, u1(C)=1, u2(A)=0, u2(B)=.1, and u2(C)=1, so the expected utility of B is .5*.5 + .5*.1 = .3 utils, and the expected utility of the lottery is .2*1 + .8*0 = .2 utils, so B is better. The result changes depending on how we define a util, even though we are modeling the same knowledge over preferences in each situation.

Anything with moral uncertainty, such as a value loading agent, needs to know how to add utility functions, not just utilitarians. I do not have a satisfactory solution to this, although I have come up with 2 attempted solutions, neither of which is entirely satisfactory.

My first idea was to normalize the standard deviation of each utility function to 1. For example, in the XYZ world, after normalizing u and v so that their values have standard deviation 1, we get (approximately) u(X)=0, u(Y)=.802, u(Z)=2.405, v(X)=0, v(Y)=2.449, v(Z)=1.225, so (u+v)(X)=0, (u+v)(Y)=3.251, and (u+v)(Z)=3.630. Z is thus declared the best option overall. However, if there are an infinite number of possible states, then this is impossible unless we have some sort of a priori probability distribution over the possible states. Even more frightening is the fact that this does not respect independence of irrelevant alternatives. Let's suppose that we find out that X is impossible. Good; no one wanted it anyway, so this shouldn't change anything, right? But if you exclude X and set Y as the 0 value for each utility function, then we get u(Y)=0, u(Z)=2, v(Y)=0, v(Z)=-2, (u+v)(Y)=0, (u+v)(Z)=0. The relative values of Y and Z in our preference aggregator changed even though all we did was exclude an option that everyone already agreed we should avoid.

Then it occurred to me that we have much more knowledge about the relative values of options that we are already quite familiar with, so it seems reasonable to assume that most of our moral uncertainty is about the value of options that we are not so familiar with. For example, in the ABC world, if you make decisions involving A and B all the time, but C is an unfamiliar option that you have not thought much about, it might be tempting to accept the first calculation, which gave B a value of 1 util and the lottery a value of 1.2 utils. This seems like a promising heuristic, but is difficult to formalize, and does not completely solve the problem. For instance, if both B and C are unfamiliar, then this heuristic does not have any advice to give.

A utility-maximizing varient of AIXI

15 AlexMennen 17 December 2012 03:48AM

Response to: Universal agents and utility functions

Related approaches: Hibbard (2012)Hay (2005)

Background

 

Here is the function implemented by finite-lifetime AI\xi:

{\displaystyle \dot{y}_{k}:=\arg\max_{y_{k}\in Y}\sum_{x_{k}\in X}\max_{y_{k+1}\in Y}\sum_{x_{k+1}\in X}...\max_{y_{m}\in Y}\sum_{x_{m}\in X}\left(r\left(x_{k}\right)+...+r\left(x_{m}\right)\right)\cdot\xi\left(\dot{y}\dot{x}_{<k}y\underline{x}_{k:m}\right)},

where m is the number of steps in the lifetime of the agent, k is the current step being computed, X is the set of possible observations, Y is the set of possible actions, r is a function that extracts a reward value from an observation, a dot over a variable represents that its value is known to be the true value of the action or observation it represents, underlines represent that the variable is an input to a probability distribution, and \xi is a function that returns the probability of a sequence of observations, given a certain known history and sequence of actions, and starting from the Solomonoff prior. More formally,

{\displaystyle \xi\left(\dot{y}\dot{x}_{<k}y\underline{x}_{k:m}\right)=\left(\sum_{q\in Q:q\left(y_{\leq m}\right)=x_{\leq m}}2^{-\ell\left(q\right)}\right)\diagup\left(\sum_{q\in Q:q\left(\dot{y}_{<k}\right)=\dot{x}_{<k}}2^{-\ell\left(q\right)}\right)},

where Q is the set of all programs, \ell is a function that returns the length of a program in bits, and a program applied to a sequence of actions returns the resulting sequence of observations. Notice that the denominator is a constant, depending only on the already known \dot{y}\dot{x}_{%3Ck}, and multiplying by a positive constant does not change the argmax, so we can pretend that the denominator doesn't exist. If q is a valid program, then any longer program with q as a prefix is not a valid program, so {\displaystyle%20\sum_{q\in%20Q}2^{-\ell\left%28q\right%29}\leq1}.

 

Problem

 

A problem with this is that it can only optimize over the input it receives, not over aspects of the external world that it cannot observe. Given the chance, AI\xi would hack its input channel so that it would only observe good things, instead of trying to make good things happen (in other words, it would wirehead itself). Anja specified a variant of AI\xi in which she replaced the sum of rewards with a single utility value and made the domain of the utility function be the entire sequence of actions and observations instead of a single observation, like so:

{\displaystyle%20\dot{y}_{k}:=\arg\max_{y_{k}\in%20Y}\sum_{x_{k}\in%20X}\max_{y_{k+1}\in%20Y}\sum_{x_{k+1}\in%20X}...\max_{y_{m}\in%20Y}\sum_{x_{m}\in%20X}U\left%28\dot{y}\dot{x}_{%3Ck}yx_{k:m}\right%29\cdot\xi\left%28\dot{y}\dot{x}_{%3Ck}y\underline{x}_{k:m}\right%29}.

This doesn't really solve the problem, because the utility function still only takes what the agent can see, rather than what is actually going on outside the agent. The situation is a little better because the utility function also takes into account the agent's actions, so it could punish actions that look like the agent is trying to wirehead itself, but if there was a flaw in the instructions not to wirehead, the agent would exploit it, so the incentive not to wirehead would have to be perfect, and this formulation is not very enlightening about how to do that.

[Edit: Hibbard (2012) also presents a solution to this problem. I haven't read all of it yet, but it appears to be fairly different from what I suggest in the next section.]

 

Solution

 

Here's what I suggest instead: everything that happens is determined by the program that the world is running on and the agent's actions, so the domain of the utility function should be Q\times%20Y^{m}. The apparent problem with that is that the formula for AI\xi does not contain any mention of elements of Q. If we just take the original formula and replace r\left(x_{k}\right)+...+r\left(x_{m}\right)

with U\left(q,\dot{y}_{<k}y_{k:m}\right), it wouldn't make any sense. However, if we expand out \xi in the original formula (excluding the unnecessary denominator), we can move the sum of rewards inside the sum over programs, like this:

{\displaystyle \dot{y}_{k}:=\arg\max_{y_{k}\in Y}\sum_{x_{k}\in X}\max_{y_{k+1}\in Y}\sum_{x_{k+1}\in X}...\max_{y_{m}\in Y}\sum_{x_{m}\in X}\sum_{q\in Q:q\left(y_{\leq m}\right)=x_{\leq m}}\left(r\left(x_{k}\right)+...+r\left(x_{m}\right)\right)2^{-\ell\left(q\right)}}.

Now it is easy to replace the sum of rewards with the desired utility function.

{\displaystyle \dot{y}_{k}:=\arg\max_{y_{k}\in Y}\sum_{x_{k}\in X}\max_{y_{k+1}\in Y}\sum_{x_{k+1}\in X}...\max_{y_{m}\in Y}\sum_{x_{m}\in X}\sum_{q\in Q:q\left(y_{\leq m}\right)=x_{\leq m}}U\left(q,\dot{y}_{<k}y_{k:m}\right)2^{-\ell\left(q\right)}}.

With this formulation, there is no danger of the agent wireheading, and all U has to do is compute everything that happens when the agent performs a given sequence of actions in a given program, and decide how desirable it is. If the range of U is unbounded, then this might not converge. Let's assume throughout this post that the range of U is bounded.

[Edit: Hay (2005) presents a similar formulation to this.]

 

Extension to infinite lifetimes

 

The previous discussion assumed that the agent would only have the opportunity to perform a finite number of actions. The situation gets a little tricky when the agent is allowed to perform an unbounded number of actions. Hutter uses a finite look-ahead approach for AI\xi, where on each step k, it pretends that it will only be performing m_{k} actions, where \forall%20k\,%20m_{k}\gg%20k.

{\displaystyle \dot{y}_{k}:=\arg\max_{y_{k}\in Y}\sum_{x_{k}\in X}\max_{y_{k+1}\in Y}\sum_{x_{k+1}\in X}...\max_{y_{m_{k}}\in Y}\sum_{x_{m_{k}}\in X}\left(r\left(x_{k}\right)+...+r\left(x_{m_{k}}\right)\right)\cdot\xi\left(\dot{y}\dot{x}_{<k}y\underline{x}_{k:m_{k}}\right)}.

If we make the same modification to the utility-based variant, we get

{\displaystyle \dot{y}_{k}:=\arg\max_{y_{k}\in Y}\sum_{x_{k}\in X}\max_{y_{k+1}\in Y}\sum_{x_{k+1}\in X}...\max_{y_{m_{k}}\in Y}\sum_{x_{m_{k}}\in X}\sum_{q\in Q:q\left(y_{\leq m_{k}}\right)=x_{\leq m_{k}}}U\left(q,\dot{y}_{<k}y_{k:m_{k}}\right)2^{-\ell\left(q\right)}}.

This is unsatisfactory because the domain of U was supposed to consist of all the information necessary to determine everything that happens, but here, it is missing all the actions after step m_{k}. One obvious thing to try is to set m_{k}:=\infty. This will be easier to do using a compacted expression for AI\xi:

{\displaystyle \dot{y}_{k}:=\arg\max_{y_{k}\in Y}\max_{p\in P:p\left(\dot{x}_{<k}\right)=\dot{y}_{<k}y_{k}}\sum_{q\in Q:q\left(\dot{y}_{<k}\right)=\dot{x}_{<k}}\left(r\left(x_{k}^{pq}\right)+...+r\left(x_{m_{k}}^{pq}\right)\right)2^{-\ell\left(q\right)}},

where P is the set of policies that map sequences of observations to sequences of actions and x_{i}^{pq} is shorthand for the last observation in the sequence returned by q\left(p\left(\dot{x}_{<k}x_{k:i-1}^{pq}\right)\right). If we take this compacted formulation, modify it to accommodate the new utility function, set m_{k}:=\infty, and replace the maximum with a supremum (since there's an infinite number of possible policies), we get

{\displaystyle \dot{y}_{k}:=\arg\max_{y_{k}\in Y}\sup_{p\in P:p\left(\dot{x}_{<k}\right)=\dot{y}_{<k}y_{k}}\sum_{q\in Q:q\left(\dot{y}_{<k}\right)=\dot{x}_{<k}}U\left(q,\dot{y}_{<k}y_{k}y_{k+1:\infty}^{pq}\right)2^{-\ell\left(q\right)}},

where y_{i}^{pq} is shorthand for the last action in the sequence returned by p\left(q\left(\dot{y}_{<k}y_{k}y_{k+1:i-1}^{pq}\right)\right).

 

But there is a problem with this, which I will illustrate with a toy example. Suppose Y:=\left\{ a,b\right\} , and U\left(q,y_{1:\infty}\right)=0 when \forall k\in\mathbb{N}\, y_{k}=a, and for any n\in\mathbb{N}, U\left(q,y_{1:\infty}\right)=1-\frac{1}{n} when y_{n}=b and \forall k<n\, y_{k}=a. (U does not depend on the program q in this example). An agent following the above formula would output a on every step, and end up with a utility of 0, when it could have gotten arbitrarily close to 1 by eventually outputting b.

 

To avoid problems like that, we could assume the reasonable-seeming condition that if y_{1:\infty} is an action sequence and \left\{ y_{1:\infty}^{n}\right\} _{n=1}^{\infty} is a sequence of action sequences that converges to y_{1:\infty} (by which I mean \forall k\in\mathbb{N}\,\exists N\in\mathbb{N}\,\forall n>N\, y_{k}^{n}=y_{k}), then {\displaystyle \lim_{n\rightarrow\infty}U\left(q,y_{1:\infty}^{n}\right)=U\left(q,y_{1:\infty}\right)}.

 

Under that assumption, the supremum is in fact a maximum, and the formula gives you an action sequence that will reach that maximum (proof below).

 

If you don't like the condition I imposed on U, you might not be satisfied by this. But without it, there is not necessarily a best policy. One thing you can do is, on step 1, pick some extremely small \varepsilon>0, pick any element from

{\displaystyle \left\{ p^{*}\in P:\sum_{q\in Q}U\left(q,y_{1:\infty}^{p^{*}q}\right)2^{-\ell\left(q\right)}>\left(\sup_{p\in P}\sum_{q\in Q}U\left(q,y_{1:\infty}^{pq}\right)2^{-\ell\left(q\right)}\right)-\varepsilon\right\} }, and then follow that policy for the rest of eternity, which will guarantee that you do not miss out on more than \varepsilon of expected utility.

 

Proof of criterion for supremum-chasing working

 

definition: If y_{1:\infty} is an action sequence and \left\{ y_{1:\infty}^{n}\right\} _{n=1}^{\infty} is an infinite sequence of action sequences, and \forall k\in\mathbb{N}\,\exists N\in\mathbb{N}\,\forall n>N\, y_{k}^{n}=y_{k}, then we say \left\{ y_{1:\infty}^{n}\right\} _{n=1}^{\infty} converges to y_{1:\infty}. If p is a policy and \left\{ p_{n}\right\} _{n=1}^{\infty} is a sequence of policies, and \forall k\in\mathbb{N}\,\forall x_{<k}\in X^{k}\,\exists N\in\mathbb{N}\,\forall n>N\, p\left(x_{<k}\right)=p_{n}\left(x_{<k}\right), then we say \left\{ p_{n}\right\} _{n=1}^{\infty} converges to p.

 

assumption (for lemma 2 and theorem): If \left\{ y_{1:\infty}^{n}\right\} _{n=1}^{\infty} converges to y_{1:\infty}, then {\displaystyle \lim_{n\rightarrow\infty}U\left(q,y_{1:\infty}^{n}\right)=U\left(q,y_{1:\infty}\right)}.

 

lemma 1: The agent described by

{\displaystyle \dot{y}_{k}:=\arg\max_{y_{k}\in Y}\sup_{p\in P:p\left(\dot{x}_{<k}\right)=\dot{y}_{<k}y_{k}}\sum_{q\in Q:q\left(\dot{y}_{<k}\right)=\dot{x}_{<k}}U\left(q,\dot{y}_{<k}y_{k}y_{k+1:\infty}^{pq}\right)2^{-\ell\left(q\right)}} follows a policy that is the limit of a sequence of policies \left\{ p_{n}\right\} _{n=1}^{\infty} such that

{\displaystyle \lim_{n\rightarrow\infty}\sum_{q\in Q}U\left(q,y_{1:\infty}^{p_{n}q}\right)2^{-\ell\left(q\right)}=\sup_{p\in P}\sum_{q\in Q}U\left(q,y_{1:\infty}^{pq}\right)2^{-\ell\left(q\right)}}.

 

proof of lemma 1: Any policy can be completely described by the last action it outputs for every finite observation sequence. Observations are returned by a program, so the set of possible finite observation sequences is countable. It is possible to fix the last action returned on any particular finite observation sequence to be the argmax, and still get arbitrarily close to the supremum with suitable choices for the last action returned on the other finite observation sequences. By induction, it is possible to get arbitrarily close to the supremum while fixing the last action returned to be the argmax for any finite set of finite observation sequences. Thus, there exists a sequence of policies approaching the policy that the agent implements whose expected utilities approach the supremum.

 

lemma 2: If p is a policy and \left\{ p_{n}\right\} _{n=1}^{\infty} is a sequence of policies converging to p, then

{\displaystyle \sum_{q\in Q}U\left(q,y_{1:\infty}^{pq}\right)2^{-\ell\left(q\right)}=\lim_{n\rightarrow\infty}\sum_{q\in Q}U\left(q,y_{1:\infty}^{p_{n}q}\right)2^{-\ell\left(q\right)}}.

 

proof of lemma 2: Let \varepsilon>0. On any given sequence of inputs x_{1:\infty}\in X^{\infty}, \left\{ p_{n}\left(x_{1:\infty}\right)\right\} _{n=1}^{\infty} converges to p\left(x_{1:\infty}\right), so, by assumption, \forall q\in Q\,\exists N\in\mathbb{N}\,\forall n\geq N\,\left|U\left(q,y_{1:\infty}^{pq}\right)-U\left(q,y_{1:\infty}^{p_{n}q}\right)\right|<\frac{\varepsilon}{2}.

For each N\in\mathbb{N}, let Q_{N}:=\left\{ q\in Q:\forall n\geq N\,\left|U\left(q,y_{1:\infty}^{pq}\right)-U\left(q,y_{1:\infty}^{p_{n}q}\right)\right|<\frac{\varepsilon}{2}\right\} . The previous statement implies that {\displaystyle \bigcup_{N\in\mathbb{N}}Q_{N}=Q}, and each element of \left\{ Q_{N}\right\} _{N\in\mathbb{N}} is a subset of the next, so

{\displaystyle \exists N\in\mathbb{N}\,\sum_{q\in Q\setminus Q_{N}}2^{-\ell\left(q\right)}<\frac{\varepsilon}{2\left(\sup U-\inf U\right)}}. The range of U is bounded, so \sup U and \inf U are defined. This also implies that the difference in expected utility, given any information, of any two policies, is bounded. More formally:{\displaystyle \forall Q'\subset Q\,\forall p',p''\in P\,\left|\left(\left(\sum_{q\in Q'}U\left(q,y_{1:\infty}^{p'q}\right)2^{-\ell\left(q\right)}\right)\diagup\left(\sum_{q\in Q'}2^{-\ell\left(q\right)}\right)\right)-\left(\left(\sum_{q\in Q'}U\left(q,y_{1:\infty}^{p''q}\right)2^{-\ell\left(q\right)}\right)\diagup\left(\sum_{q\in Q'}2^{-\ell\left(q\right)}\right)\right)\right|\leq\sup U-\inf U},

so in particular,

{\displaystyle \left|\left(\sum_{q\in Q\setminus Q_{N}}U\left(q,y_{1:\infty}^{pq}\right)2^{-\ell\left(q\right)}\right)-\left(\sum_{q\in Q\setminus Q_{N}}U\left(q,y_{1:\infty}^{p_{N}q}\right)2^{-\ell\left(q\right)}\right)\right|\leq\left(\sup U-\inf U\right)\sum_{q\in Q\setminus Q_{N}}2^{-\ell\left(q\right)}<\frac{\varepsilon}{2}}.

{\displaystyle \left|\left(\sum_{q\in Q}U\left(q,y_{1:\infty}^{pq}\right)2^{-\ell\left(q\right)}\right)-\left(\sum_{q\in Q}U\left(q,y_{1:\infty}^{p_{N}q}\right)2^{-\ell\left(q\right)}\right)\right|\leq\left|\left(\sum_{q\in Q_{N}}U\left(q,y_{1:\infty}^{pq}\right)2^{-\ell\left(q\right)}\right)-\left(\sum_{q\in Q_{N}}U\left(q,y_{1:\infty}^{p_{N}q}\right)2^{-\ell\left(q\right)}\right)\right|+\left|\left(\sum_{q\in Q\setminus Q_{N}}U\left(q,y_{1:\infty}^{pq}\right)2^{-\ell\left(q\right)}\right)-\left(\sum_{q\in Q}U\left(q,y_{1:\infty}^{p_{N}q}\right)2^{-\ell\left(q\right)}\right)\right|<\frac{\varepsilon}{2}+\frac{\varepsilon}{2}=\varepsilon}.

 

theorem:

{\displaystyle \sum_{\dot{q}\in Q}U\left(\dot{q},\dot{y}_{1:\infty}\right)2^{-\ell\left(\dot{q}\right)}=\sup_{p\in P}\sum_{q\in Q}U\left(q,y_{1:\infty}^{pq}\right)2^{-\ell\left(q\right)}},

where {\displaystyle \dot{y}_{k}:=\arg\max_{y_{k}\in Y}\sup_{p\in P:p\left(\dot{x}_{<k}\right)=\dot{y}_{<k}y_{k}}\sum_{q\in Q:q\left(\dot{y}_{<k}\right)=\dot{x}_{<k}}U\left(q,\dot{y}_{<k}y_{k}y_{k+1:\infty}^{pq}\right)2^{-\ell\left(q\right)}}.

 

proof of theorem: Let's call the policy implemented by the agent p^{*}.

By lemma 1, there is a sequence of policies \left\{ p_{n}\right\} _{n=1}^{\infty}

converging to p^{*} such that

{\displaystyle \lim_{n\rightarrow\infty}\sum_{q\in Q}U\left(q,y_{1:\infty}^{p_{n}q}\right)2^{-\ell\left(q\right)}=\sup_{p\in P}\sum_{q\in Q}U\left(q,y_{1:\infty}^{pq}\right)2^{-\ell\left(q\right)}}.

By lemma 2,

{\displaystyle \sum_{q\in Q}U\left(q,y_{1:\infty}^{p^{*}q}\right)2^{-\ell\left(q\right)}=\sup_{p\in P}\sum_{q\in Q}U\left(q,y_{1:\infty}^{pq}\right)2^{-\ell\left(q\right)}}.


Meetup : Berkeley meetup: choice blindness

1 AlexMennen 25 September 2012 05:31AM

Discussion article for the meetup : Berkeley meetup: choice blindness

WHEN: 26 September 2012 07:00:00PM (-0700)

WHERE: Berkeley, CA

This week's meetup will be at Zendo, as usual. The discussion topic will be choice blindness, particularly the recent experiment showing that choice blindness applies to moral positions (http://www.nature.com/news/how-to-confuse-a-moral-compass-1.11447), and what this means for good epistemic practice. Recent Less Wrong discussion on the matter: http://lesswrong.com/lw/elg/new_study_on_choice_blindness_in_moral_positions/ For directions to Zendo, see the mailing list http://groups.google.com/group/bayarealesswrong or call me at four-zero-eight-nine-six-six-nine-two-seven-four.

Discussion article for the meetup : Berkeley meetup: choice blindness

View more: Next