All of Toby_Ord's Comments + Replies

Thanks — this looks promising.

One thing I noticed is that there is an interesting analogy between your model and a fairly standard model in economics where society consists of a representative agent in each time period (representing something like a generation, but without overlap) each trying to maximise its own utility. They can plan based on the utilities of subsequent generations (e.g. predicting that the next generation will undo this generation's policies on some topic) but they don't inherently value those utilities. This is then understood via the ... (read more)

4Rohin Shah
Indeed this seems related! We'd hope that there isn't too much conflict between the different "generations", since the intent is for the overseer to enable long-term plans, but it seems like something that could happen if the overseer is bad enough.

This is an interesting theorem which helps illuminate the relationship between unbounded utilities and St Petersburg gambles. I particularly appreciate that you don't make an explicit assumption that the values of gambles must be representable by real numbers which is very common, but unhelpful in a setting like this. However, I do worry a bit about the argument structure.

The St Petersburg gamble is a famously paradox-riddled case. That is, it is a very difficult case where it isn't clear what to say, and many theories seem to produce outlandish results. W... (read more)

4MichaelStJules
I think the more important takeaway is that the (countable) sure thing principle and transitivity together rule out preferences allowing St. Petersburg-like lotteries, and so "unbounded" preferences. I recommend https://onlinelibrary.wiley.com/doi/abs/10.1111/phpr.12704 It discusses more ways preferences allowing St. Petersburg-like lotteries seem irrational, like choosing dominated strategies, dynamic inconsistency and paying to avoid information. Furthermore, they argue that the arguments supporting the finite Sure Thing Principle are generally also arguments in favour of the Countable Sure Thing Principle, because they don't depend on the number of possibilities in a lottery being finite. So, if you reject the Countable Sure Thing Principle, you should probably reject the finite one, too, and if you accept St. Petersburg-like lotteries, you need to in principle accept behaviour that seems irrational. They also have a general vNM-like representation theorem, dropping the Archimedean/continuity axiom, and replacing the Independence axiom with Countable Independence, and with transitivity and completeness, they get utility functions with values in lexicographically ordered ordinal sequences of bounded real utilities. (They say the sequences can have any ordinal to order them, but that seems wrong to me, since I'd think infinite length lexicographically ordered sequences get you St. Petersburg-like lotteries and violate Limitedness, but maybe I'm misunderstanding. EDIT: I think they meant you can have a an infinite sequence of dominated components, not an infinite sequence of dominating components, so you check the most important component first, and then the second, and continue for possibly infinitely many. Well-orderedness ensures there's always a next one to check.)
5paulfchristiano
I think that weak outcome-lottery dominance is inconsistent with transitivity + unbounded utilities in both directions (or unbounded utilities in one direction + the sure thing principle), rather than merely producing strange results. Though we could summarize "violates weak outcome-lottery dominance" as a strange result. Violating weak outcome-lottery dominance means that a mix of gambles, each strictly better than a particular outcome X, can fail to be at least as good as X. If you give up on this property, or on transitivity, then even if you are assigning numbers you call "utilities" to actions I don't think it's reasonable to call them utilities in the decision-theoretic sense, and I'm comfortable saying that your procedure should no longer be described as "expected utility maximization." So I'd conclude that there simply don't exist any preferences represented by unbounded utility functions (over the space of all lotteries), and that there is no patch to the notion of utility maximization that fixes this problem without giving up on some defining feature of EU maximization. There may nevertheless be theories that are well-described as maximizing an unbounded utility function in some more limited situations. And there may well be preferences over a domain other than lotteries which are described intuitively by an unbounded utility function. (Though note that if you are only considering lotteries over a finite space then your utility function is necessarily bounded.) And although it seems somewhat less likely it could also be that in retrospect I will feel I was wrong about the defining features of EU maximization, and mixing together positive lotteries to get a negative lottery is actually consistent with its spirit. I think it's also worth observing that although St Petersburg cases are famously paradox-riddled, these cases seem overwhelmingly important on a conventional utilitarian view even before we consider any exotic hypotheses. Indeed, I personally b

Regarding your question, I don't see theoretical reasons why one shouldn't be making deals like that (assuming one can and would stick to them etc). I'm not sure which decision theory to apply to them though.

The Moral Parliament idea generally has a problem regarding time. If it is thought of as making decisions for the next action (or other bounded time period), with new distribution of votes etc when the next choice comes up, then there are intertemporal swaps (and thus pareto improvements according to each theory) that it won't be able to achieve. This is pretty bad, as it at least appears to be getting pareto dominated by another method. However, if it is making one decision for all time over all policies for resolving future decisions, then (1) it is even... (read more)

This is a good idea, though not a new one. Others have abandoned the idea of a formal system for this on the grounds that:

1) It may be illegal 2) Quite a few people think it is illegal or morally dubious (whether or not it is actually illegal or immoral)

It would be insane to proceed with this without confirming (1). If illegal, it would open you up to criminal prosecution, and more importantly, seriously hurt the movements you are trying to help. I think that whether or not it turns out to be illegal, (2) is sufficient reason to not pursue it. It may cause... (read more)

0blogospheroid
Thanks, Toby. I expected that the legal risks would be quite an issue. Point noted. I had not expected this to be a new idea as well, after all it seemed too simple. I guess a more informal means is good for now. Hope the EA forum has such a place to make this discussion.
Toby_Ord110

This is a really nice and useful article. I particularly like the list of problems AI experts assumed would be AI-complete, but turned out not to be.

I'd add that if we are trying to reach the conclusion that "we should be more worried about non-general intelligences than we currently are", then you don't need it to be true that general intelligences are really difficult. It would be enough that "there is a reasonable chance we will encounter a dangerous non-general one before a dangerous general one". I'd be inclined to believe that ev... (read more)

Toby_Ord160

Thanks for bringing this up Luke. I think the term 'friendly AI' has become something of an albatross around our necks as it can't be taken seriously by people who take themselves seriously. This leaves people studying this area without a usable name for what they are doing. For example, I talk with parts of the UK government about the risks of AGI. I could never use the term 'friendly AI' in such contexts -- at least without seriously undermining my own points. As far as I recall, the term was not originally selected with the purpose of getting traction w... (read more)

This is quite possibly the best LW comment I've ever read. An excellent point with a really concise explanation. In fact it is one of the most interesting points I've seen within Kolmogorov complexity too. Well done on independently deriving the result!

1Oscar_Cunningham
Thanks!

Without good ways to overcome selection bias, it is unclear that data like this can provide any evidence of outsized impact of unconventional approaches. I would expect a list of achievements as impressive as the above whether or not there was any correlation between the two.

Carl,

You are completely right that there is a somewhat illicit factor-of-1000 intuition pump in a certain direction in the normal problem specification, which makes it a bit one-sided. Will McAskill and I had half-written a paper on this and related points regarding decision-theoretic uncertainty and Newcomb's problem before discovering that Nozick had already considered it (even if very few people have read or remembered his commentary on this).

We did still work out though that you can use this idea to create compound problems where for any reasonable di... (read more)

0CarlShulman
Thanks, I'll ask him for a copy.

Regarding (2), this is a particularly clean way to do it (with some results of my old simulations too).

http://www.amirrorclear.net/academic/papers/sipd.pdf http://www.amirrorclear.net/academic/ideas/dilemma/index.html

Toby_Ord110

We can't use the universal prior in practice unless physics contains harnessable non-recursive processes. However, this is exactly the situation in which the universal prior doesn't always work. Thus, one source of the 'magic' is through allowing us to have access to higher levels of computation than the phenomena we are predicting (and to be certain of this).

Also, the constants involved could be terrible and there are no guarantees about this (not even probabilistic ones). It is nice to reach some ratio in the limit, but if your first Graham's number of guesses are bad, then that is very bad for (almost) all purposes.

3Wei Dai
My position is that the uncomputabiilty of the universal prior shouldn't count against it. I think the fact that it works so well shows that our actual prior is likely also uncomputable, and that means we have to handle uncomputable priors in our decision theory, for example by specifying that we choose the option that we can prove (or just heuristically believe) has the highest expected payoff, instead of actually computing the expected payoffs. A worse problem is that there seems to be reason to think that our actual prior is not just uncomputable, but unformalizable. See my earlier posts on this.
0Jess_Riedel
Could you suggest a source for further reading on this?
Toby_Ord190

Phil,

It's not actually that hard to make a commitment to give away a large fraction of your income. I've done it, my wife has done it, several of my friends have done it etc. Even for yourself, the benefits of peace of mind and lack of cognitive dissonance will be worth the price, and by my calculations you can make the benefits for others at least 10,000 times as big as the costs for yourself. The trick is to do some big thinking and decision making about how to live very rarely (say once a year) then limit your salary through regular giving. That way you... (read more)

-7denisbider
7MichaelVassar
Toby, ignoring donations to SIAI and possibly FHI I'm still very skeptical of your claims. GiveWell have done analysis strongly indicating that the cheapest lives to save actually cost between $1K and $2K, but one would have to search for a long time to find them GiveWell and much longer to do GiveWell's analysis yourself. Evaluating GiveWell is intermediate and most people lack the cognitive abilities to do that. Furthermore, the lives in question are fairly low value compared to our own lives. I don't have any qualms in saying that if purely selfish I'd unhesitatingly play 5 full chamber Russian Roulette rather than being economically, physically, and mentally reduced to the conditions of a typical Tuberculosis victim regardless of what happiness researchers may say about them. Note that I have lived in the 3rd world and have known such people so it's not just distance that makes me say that. I have some feel for the odds against snake eyes and with more hesitation I'd go for that too. In any event I have more feel for that then I do for what giving up essentially all my human capital would mean from the inside. Anyway, based on the numbers I just gave, saving a quality of life comparable to my own would cost more like $50K. Would I spend $50K to save my life? Hell yes. To avoid a 1% chance of death? Maybe. Lets try that again like a behavioral economist. To reduce my chance of death in the next 10 years by half? Not so sure. I'm a 31 year old male so ignoring other considerations that would constitute a 1% risk of death. http://www.ssa.gov/OACT/STATS/table4c6.html Other considerations probably halve it already so make it 15 years and it's still borderline. Though inclined to consider it somewhat for altruistic reasons, I don't pay for cryonics, which is pretty much pure selfish survival along the above lines and which would be considerably cheaper. This leads me to conclude that I would have to be over 1% altruistic to spend on third world aid, not .01% as you
5brazil84
Toby, I am curious: How many children do you have or plan to have? Couldn't one argue that the expense of having a child in the West is like buying 100,000 Starbuck's lattes?

I didn't watch the video, but I don't see how that could be true. Occam's razor is about complexity, while the conjunction fallacy is about logical strength.

Sure 'P & Q' is more complex than 'P', but 'P' is simpler than '(P or ~Q)' despite it being stronger in the same way (P is equivalent to (P or ~Q) & (P or Q)).

(Another way to see this is that violating Occam's razor does not make things fallacies).

6timtyler
The actual quote is: "...which of course violates the conjunction rule of probability theory - also known as Occam's razor - which says that a more complicated event cannot be more probable than an a strictly simpler event that includes the more complicated one." 41 minutes in.

This certainly doesn't work in all cases:

There is a hidden object which is either green, red or blue. Three people have conflicting opinions about its colour, based on different pieces of reasoning. If you are the one who believes it is green, you have to add up the opponents who say not-green, despite the fact that there is no single not-green position (think of the symmetry -- otherwise everyone could have too great confidence). The same holds true if these are expert opinions.

The above example is basically as general as possible, so in order for your ar... (read more)

4JamesAndrix
In practice all arguments will share some premises and some conclusions, in messy asymmetrical ways. If the not-greens share a a consistent rationale about why the object cannot be green, then I need to take that into account. If the red supporter contends that all green and blue objects were lost in the color wars, while the blue supporter contends that all objects are fundamentally blue and besides the color wars never happened, then their opinions roughly cancel each other out. (Barring other reasons for me to view one as more rational than the other.) I suspect that there are things to be said about islam that both atheists and christians would agree on. That's a block that a rational muslim should take into account. Our disagreeing conclusions about god are secondary. If I'm going to update my position because 56% of experts agree on something, then I want to know what I'm going to update to.

I don't think I can persuaded.

I have many good responses to the comments here, and I suppose I could sketch out some of the main arguments against anti-realism, but there are also many serious demands on my time and sadly this doesn't look like a productive discussion. There seems to be very little real interest in finding out more (with a couple of notable exceptions). Instead the focus is on how to justify what is already believed without finding out any thing else about what the opponents are saying (which is particularly alarming given that many commenters are pointing out that they don't understand what the opponents are saying!).

Given all of this, I fear that writing a post would not be a good use of my time.

6CarlShulman
Alas. Perhaps some Less Wrongers with more time will write and post a hypothetical apostasy. I invite folk to do so.
1DonGeddis
I, for one, am interested in hearing arguments against anti-realism. If you don't have personal interest in writing up a sketch, that's fine. Might you have some links to other people who have already done so?
4Paul Crowley
This is a little unfair; as soon as you take a deflationary stance on anything, you're saying that the other stance doesn't really have comprehensible content, and it's a mistake to turn that into a general-purpose dismissal of deflationary stances. If you think that's more true here than it is in other discussion forums, we're doing something very wrong. I understand that you're not able to spend time writing for this audience, but for those of us who do want to find out more about what moral realists are saying, every link you can provide to existing essays is valuable.

You are correct that it is reasonable to assign high confidence to atheism even if it doesn't have 80% support, but we must be very careful here. Atheism is presumably the strongest example of such a claim here on Less Wrong (i.e. one which you can tell a great story why so many intelligent people would disagree etc and hold a high confidence in the face of disagreement). However, this does not mean that we can say that any other given view is just like atheism in this respect and thus hold beliefs in the face of expert disagreement, that would be far too convenient.

0komponisto
Of course not; the substance of one's reasons for disagreeing matters greatly. In this case, I suspect there's probably a significant amount of correlation/non-independence between the reasons for believing atheism and believing something like moral non-realism. One thing we should take away from cases like atheism is that surveys probably shouldn't be interpreted naively, but rather as somewhat noisy information. I think my own heuristic (on binary questions where I already have a strong opinion) is basically to look on which side of 50% my position falls; if the majority agrees with me (or, say, the average confidence in my position is over 50%), I tend to regard that as (more) evidence in my favor, with the strength increasing as the percentage increases. (This, I think, would be part of how I would answer Yvain.)
9CarlShulman
Strong agreement about not overgeneralizing. It does appear, however, that libertarianism about free well, non-physicalism about the mind, and a number of sorts of moral realism form a cluster, sharing the feature of reifying certain concepts in our cognitive algorithms even when they can be 'explained away.' Maybe we can discuss this tomorrow night.

Roko, you make a good point that it can be quite murky just what realism and anti-realism mean (in ethics or in anything else). However, I don't agree with what you write after that. Your Strong Moral Realism is a claim that is outside the domain of philosophy, as it is an empirical claim in the domain of exo-biology or exo-sociology or something. No matter what the truth of a meta-ethical claim, smart entities might refuse to believe it (the same goes for other philosophical claims or mathematical claims).

Pick your favourite philosophical claim. I'm sure ... (read more)

0Roko
There are clearly some examples where there can be interesting things to say that aren't really empirical, e.g. decision theory, mystery of subjective experience. But I think that this isn't one of them. Suffice it to say I can't think of anything that makes the debate between weak realism and antirealism at all interesting or worthy of attention. Certainly, Friendly AI theorists ought not care about the difference, because the empirical claims about an AI system will do are identical. Once the illusions and fallacies surrounding rationalist moral psychology has been debunked, proponents of other AI motivation methods than FAI also ought not to care about the weak realism vs. anti-realism pseudo-question
7Roko
If there's a philosophical claim that intelligent agents across the universe wouldn't display massive agreement on, then I don't really think it is worth its salt. I think that this principle can be used to eliminate a lot of nonsense from philosophy. Which of anti-realism or weak realism is true seems to be a question we can eliminate. Whether strong realism is true or not seems substantive, because it matters to our policy which is true.
0[anonymous]
I'm having trouble reconciling this with the beginning of your first comment:

Thanks for looking that up Carl -- I didn't know they had the break-downs. This is the more relevant result for this discussion, but it doesn't change my point much. Unless it was 80% or so in favour of anti-realism, I think holding something like 95% credence in anti-realism this is far too high for non-experts.

Atheism doesn't get 80% support among philosophers, and most philosophers of religion reject it because of a selection effect where few wish to study what they believe to be non-subjects (just as normative and applied ethicists are more likely to reject anti-realism).

You are entirely right that the 56% would split up into many subgroups, but I don't really see how this weakens my point: more philosophers support realist positions than anti-realist ones. For what its worth, the anti-realists are also fragmented in a similar way.

Disagreeing positions don't add up just because they share a feature. On the contrary, If people offer lots of different contradictory reasons for a conclusion (even if each individual has consistent beliefs) it is a sign that they are rationalizing their position.

If 2/3's of experts support proposition G , 1/3 because of reason A while rejecting B, and 1/3 because of reason B while rejecting A, and the remaining 1/3 reject A and B; then the majority Reject A, and the majority Reject B. G should not be treated as a reasonable majority view.

This should be ... (read more)

Toby_Ord130

In metaethics, there are typically very good arguments against all known views, and only relatively weak arguments for each of them. For anything in philosophy, a good first stop is the Stanford Encyclopedia of Philosophy. Here are some articles on the topic at SEP:

I think the best book to read on metaethics is:

6Wei Dai
Toby, I read through those SEP articles but couldn't find the good arguments against anti-realism that you mentioned. In contrast, the article on deontology laid out the arguments for and against it very clearly. Can you please point us more specifically to the arguments that you find persuasive? Maybe just give us some page numbers in the book that you referenced? Most of us don't really have the time to read something like that cover to cover in search of a few nuggets of information.
5Paul Crowley
Do you have a personal favourite argument against moral anti-realism in there you could point me to?
2Paul Crowley
Thanks! There were several points in your PhD thesis where I couldn't work out how to square your position with moral anti-realism - I guess I know why now :-)
Toby_Ord210

There are a lot of posts here that presuppose some combination of moral anti-realism and value complexity. These views go together well: if value is not fundamental, but dependent on characteristics of humans, then it can derive complexity from this and not suffer due to Occam's Razor.

There are another pair of views that go together well: moral realism and value simplicity. Many posts here strongly dismiss these views, effectively allocating near-zero probability to them. I want to point out that this is a case of non-experts being very much at odds with ... (read more)

8TruePath
The right response to moral realism isn't to dispute it's truth but to simply observe you don't understand the concept. I mean imagine someone started going around insisting some situations were Heret and others were Grovic but when asked to explain what made a situation Heret or Grovic he simply shrugged and said they were primitive concepts. But you persist and after observing his behavior for a period of time you work out some principle that perfectly predicts which category he will assign a given situation to, even counterfactually but when you present the algorithm to him and ask, "Ohh so is it satisfying this principle that makes one Heret rather than Grovic?" he insists that while your notion will always agrees with his notion that's not what he means. Moreover, he insists that no definition in terms of physical state could capture these concepts. Confused you press him and he says that there are special things which we can't casually interact with that determine Heret or Grovic status. Bracketing your skepticism you ask him to say what properties these new ontological objects must have. After listing a couple he adds that most importantly they can't just be random things with this structure but they also have to be Heret making or Grovic making and that's what distingushes them from all the other casually inaccessible things out there that might otherwise yield some slightly different class of things as Heret and Grovic. Frustrated you curse the guy saying he hasn't really told you anything since you didn't know what it meant to be Heret or Grovic in the first place so you surely don't know what it means to be Heret making or Grovic making. The man's reply is simply to shrug and say, "well it's a fundamental concept, if you don't understand I can't explain it to you anymore than I could explain the perceptual experience of redness to a man who had never experienced color." ---------------------------------------- In such a situation the only thing you c
3Stuart_Armstrong
It depends on the expertise; for instance, if we're talking about systems of axioms, then mathematicians may be those with the most relevant opinions as to whether one system has preference over others. And the idea that a unique system of moral axioms would have preference over all others makes no mathematical sense. If philosphers were espousing the n-realism position ("there are systems of moral axioms that are more true than others, but there will probably be many such systems, most mutually incompatible"), then I would have a hard time arguing against this. But, put quite simply, I dismiss the moral realistic position for the moment as the arguments go like this: * 1) There are moral truths that have special status; but these are undefined, and it is even undefined what makes them have this status. * 2) These undefined moral truths make a consistent system. * 3) This system is unique, according to criteria that are also undefined. * 4) Were we to discover this system, we should follow it, for reasons that are also undefined. There are too many 'undefined's in there. There is also very little philosphical literature I've encountred on 2), 3) and 4), which is at least as important as 1). A lot of the literature on 1) seems to be reducible to linguistic confusion, and (most importantly) different moral realists have different reasons for believing 1), reasons that are often contradictory. From a outsider's perspective, these seem powerful reasons to assume that philosphers are mired in confusion on this issue, and that their opinions are not determining. My strong mathematical reasons for claiming that there is no "superiority total ordering" on any general collection of systems of axioms clinches the argument for me, pending further evidence.
0taw
I don't see in what meaningful sense these people are "experts".
6Wei Dai
I accept this may be a case of the Popularization Bias (speaking for myself). I'd like to see some posts on the arguments against anti-realism...
6Paul Crowley
This point has given me a lot of pause, so forgive me my many replies. Part of the problem is that even if I were only 60% confident of moral anti-realism, I would still act on it as if I were 100% confident because I don't understand moral realism at all, and my 60% confidence is in the belief that no-one else does either.

Many posts here strongly dismiss [moral realism and simplicity], effectively allocating near-zero probability to them. I want to point out that this is a case of non-experts being very much at odds with expert opinion and being clearly overconfident. [...] For non-experts, I really can't see how one could even get to 50% confidence in anti-realism, much less the kind of 98% confidence that is typically expressed here.

One person's modus ponens is another's modus tollens. You say that professional philosophers' disagreement implies that antirealists shoul... (read more)

2[anonymous]
Looking further through the philpapers data, a big chunk of the belief in moral realism seem to be coupled with theism, where anti-realism is coupled with atheism and knowledge of science. The more a field is taught at Catholic or other religious colleges (medieval philosophy, bread-and-butter courses like epistemology and logic) the more moral realism, while philosophers of science go the other way. Philosophers of religion are 87% moral realist, while philosophers of biology are 55% anti-realist. In general, only 61% of respondents "accept" rather than lean towards atheism, and a quarter don't even lean towards atheism. Among meta-ethics specialists, 70% accept atheism, indicating that atheism and subject knowledge both predict moral anti-realism. If we restricted ourselves to the 70% of meta-ethics specialists who also accept atheism I would bet at at least 3:1 odds that moral anti-realism comes out on top. Since the Philpapers team will be publishing correlations between questions, such a bet should be susceptible to objective adjudication within a reasonable period of time. A similar pattern shows up for physicalism. In general, those interquestion correlations should help pinpoint any correct contrarian cluster.
4mattnewport
My impression of academic philosophers is that their 'expertise' is primarily in knowledge of what other philosophers have said and in the forms of academic philosophical argument. It is not expertise in true facts about the world. In other words, I would defer to their expertise on the technical details of academically accepted definitions of philosophical terms, or on the writings of Kant, much as I would defer to an expert in literary criticism on the details of what opinions other literary critics have expressed. In neither case however do I consider their opinions to be particularly relevant to the pursuit of true facts about the world. The fact that the survey you link finds 27% of philosophers 'accept or lean towards non-physicalism' increases my confidence in the above thesis.

Among target faculty listing meta-ethics as their area of study moral realism's lead is much smaller: 42.5% for moral realism and 38.2% against.

Looking further through the philpapers data, a big chunk of the belief in moral realism seems to be coupled with theism, where anti-realism is coupled with atheism and knowledge of science. The more a field is taught at Catholic or other religious colleges (medieval philosophy, bread-and-butter courses like epistemology and logic) the more moral realism, while philosophers of science go the other way. Philosophers... (read more)

Roko210

Toby, I spent a while looking into the meta-ethical debates about realism. When I thought moral realism was a likely option on the table, I meant:

Strong Moral Realism: All (or perhaps just almost all) beings, human, alien or AI, when given sufficient computing power and the ability to learn science and get an accurate map-territory distinction, will agree on what physical state the universe ought to be transformed into, and therefore they will assist you in transforming it into this state.

But modern philosophers who call themselves "realists" don... (read more)

I am a moral cognitivist. Statements like "ceteris paribus, happiness is a good thing" have truth-values. Such moral statements simply are not compelling or even interesting enough to compute the truth-value of to the vast majority of agents, even those which maximize coherent utility functions using Bayesian belief updating (that is, rational agents) or approximately rational agents.

AFAICT the closest official term for what I am is "analytic descriptivist", though I believe I can offer a better defense of analytic descriptivism than ... (read more)

From your SEP link on Moral Realism: "It is worth noting that, while moral realists are united in their cognitivism and in their rejection of error theories, they disagree among themselves not only about which moral claims are actually true but about what it is about the world that makes those claims true. "

I think this is good cause for breaking up that 56%. We should not take them as a block merely because (one component of) their conclusions match, if their justifications are conflicting or contradictory. It could still be the case that 90% of... (read more)

-2jhuffman
Is there a reason I should care about the % of any group of people that think this or that? Just give us the argument, or write another article about it. It sounds interesting.
5whpearson
Can you give pointers to prominent naturalist realists?
8Paul Crowley
Could you direct us to the best arguments for moral realism, or against anti-realism? Thanks!

Why?

Whenever you deviate from maximizing expected value (in contexts where this is possible) you can normally find examples where this behaviour looks incorrect. For example, we might be value-pumped or something.

(And why do you find it odd, BTW?)

For one thing, negentropy may well be one of the most generally useful resources, but it seems somewhat unlikely to be intrinsically good (more likely it matters what you do with it). Thus, the question looks like one of descriptive uncertainty, just as if you had asked about money: uncertainty about whethe... (read more)

2Wei Dai
Can a bargaining solution be value-pumped? My intuition says if it can, then the delegates would choose a different solution. (This seems like an interesting question to look into in more detail though.) But doesn't your answer also argue against using the bargaining solution in moral uncertainty, and in favor of just sticking with expected utility maximization (and throwing away other incompatible moral philosophies that might be value-pumped)? But what I do with negentropy largely depends on what I value, which I don't know at this point...

Thanks for the post Wei, I have a couple of comments.

Firstly, the dichotomy between Robin's approach and Nick and mine is not right. Nick and I have always been tempted to treat moral and descriptive uncertainty in exactly the same way insofar as this is possible. However, there are cases where this appears to be ill-defined (eg how much happiness for utilitarians is worth breaking a promise for Kantians?), and to deal with these cases Nick and I consider methods that are more generally applicable. We don't consider the bargaining/voting/market approach to... (read more)

3Wei Dai
Thanks for the clarifications. It looks like I might be more of a proponent for the bargaining approach than you and Nick are at this point. I think bargaining, or some of the ideas in bargaining theory (or improvements upon them), could be contenders for the canonical way of merging values (if not moral philosophies). Why? (And why do you find it odd, BTW?) I was implicitly assuming that this is the only decision (there are no future decisions), in which case the solution Nick described in his Overcoming Bias post does pick project B with certainty, I think. I know this glosses over some subtleties in your ideas, but my main goal was to highlight the difference between bargaining and linearly combining utility functions. ETA: Also, if we make the probability of the sqrt utility function much smaller, like 10^-10, then the sqrt representative has very little chance of offering enough concessions on future decisions to get its way on this one, but it would still be the case that EU(A)>EU(B).

Not quite. It depends on your beliefs about how the calculation could go wrong and how much this would change the result. If you are very confident in all parts except a minor correcting term, and are simply told that there is an error in the calculation, then you can still have some kind of rough confidence in the result (you can see how to spell this out in maths). If you know the exact part of the calculation that was mistaken, then the situation is slightly different, but still not identical to reverting to your prior.

Toby_Ord110

You may wish to check out the paper we wrote at the FHI on the problem of taking into account mistakes in one's own argument. The mathematical result is the same as the one here, but the proof is more compelling. Also, we demonstrate that when applied to the LHC, the result is very different to the above analysis.

http://www.fhi.ox.ac.uk/__data/assets/pdf_file/0006/4020/probing-the-improbable.pdf

0RobinZ
I haven't read the paper through, but the similarity in algebra cannot be denied. I have added a reference to the post.
0Paul Crowley
Thanks, interesting read. Could you expand more on the points of similarity and difference between your argument and RobinZ's? They currently seem very disparate approaches to me.
Toby_Ord150

We talk about this a bit at FHI. Nick has written a paper which is relevant:

http://www.nickbostrom.com/papers/experience.pdf

6Wei Dai
Yes, I should have remembered that paper, especially since Nick acknowledges me (along with Hal and Toby and others) in it. Do you think our preference for A (speaking for those who do prefer A) is entirely accounted for by a (perhaps partial) belief in what Nick calls Unification? Or is there also an ethical element that says two identical streams of qualia is not twice as valuable as one? Do you know of any published or unpublished ideas that deal with that question?

Everybody knows blue/green are correct categories, while grue/bleen are not.

Philosophers invented grue/bleen in order to be obviously incorrect categories, yet difficult to formally separate from the intuitively correct ones. There are of course less obvious cases, but the elucidation of the problem required them to come up with a particularly clear example.

You can also listen to an interview with one of Sarah Lichtenstein's subjects who refused to make his preferences consistent even after the money-pump aspect was explained:

http://www.decisionresearch.org/publications/books/construction-preference/listen.html

4Douglas_Knight
That is an incredible interview. Admitting that the set of preferences is inconsistent, but refusing to fix it is not so bad a conclusion - maybe he'd just make it worse (eg, by raising the bid on B to 550). At times he seems to admit that the overall pattern is irrational ("It shows my reasoning process isn't too good"). At other times, he doesn't admit the problem, but I think you're too harsh on him in framing it as refusal. I may be misunderstanding, but he seems to say that the game doesn't allow him to bid higher than 400 on B. If he values B higher than 400 (yes, an absurd mistake), but sells it for 401, merely because he wasn't allowed to value it higher, then that seems to me to be the biggest mistake. It fits the book's title, though. Maybe he just means that his sense of math is that the cap should be 400, which would be the lone example of math helping him. He seems torn between authority figures, the "rationality" of non-circular preferences and the unnamed math of expected values. I'm somewhat surprised that he doesn't see them as the same oracle. Maybe he was scarred by childhood math teachers, and a lone psychologist can't match that intimidation?
1tut
That sounds to me as though he is using expected utility to come up with his numbers, but doesn't understand expected utility, so when asked which he prefers he uses some other emotional system.

Calling fuzzy logic "truth functional" sounds like you're changing the semantics;

'Truth functional' means that the truth value of a sentence is a function of the truth values of the propositional variables within that sentence. Fuzzy logic works this way. Probability theory does not. It is not just that one is talking about degrees of truth and the other is talking about probabilities. The analogue to truth values in probability theory are probabilities, and the probability of a sentence is not a function of the probabilities of the variables ... (read more)

0[anonymous]
Eep, maybe I should edit my post so it doesn't say "fuzzy logic". Not that I know that non-truth-functional fuzzy logic is a good idea; I simply don't know that it isn't.

You can select your "fuzzy logic" functions (the set of functions used to specify a fuzzy logic, which say what value to assign A and B, A or B, and not A, as a function of the values of A and B) to be consistent with probability theory, and then you'll always get the same answer as probability theory.

How do you do this? As far as I understand, it is impossible since probability is not truth functional. For example, suppose A and B both have probability 0.5 and are independent. In this case, the probability of 'A^B' is 0.25, while the probabil... (read more)

0PhilGoetz
Calling fuzzy logic "truth functional" sounds like you're changing the semantics; but nobody really changes the semantics when they use these systems. Fuzzy logic use often becomes a semantic muddle, with people making the values simultaneously mean truth, probability, and measurement; interpreting them in an ad-hoc manner. You can tell your truth-functional logic that A^A = A. Or, you can tell it that P(A|A) = 1, so that p(A^A) = p(A).

I agree that there are very interesting questions here. We have quite natural ways of describing uncomputable functions very far up the arithmetical hierarchy, and it seems that they can be described in some kind of recursive language even if the things they describe are not recursive (using recursive in the recursion theory sense both times). Turing tried something like this in Systems of Logic Based on Ordinals (Turing, 1939), but that was with formal logic and systems where you repeatedly add the Godel sentence of a system into the system as an axiom, r... (read more)

1Wei Dai
And then there are objects that are completely outside the arithmetical hierarchy, but we probably shouldn't assign zero priors to either. Things like large cardinals, perhaps. Another mystery is, why did evolution create minds capable of thinking about these issues, given that agents equipped with a fixed UTM-based prior would have done perfectly fine in our place, at least up to now?

I outlined a few more possibilities on Overcoming Bias last year:

There are many ways Omega could be doing the prediction/placement and it may well matter exactly how the problem is set up. For example, you might be deterministic and he is precalculating your choice (much like we might be able to do with an insect or computer program), or he might be using a quantum suicide method, (quantum) randomizing whether the million goes in and then destroying the world iff you pick the wrong option (This will lead to us observing him being correct 100/100 times assu... (read more)