It’s always good news when someone else develops an idea independently from you.  It's a sign you might be onto something.  Which is why I was excited to discover that Alan Carter, Professor Emeritus of the University of Glasgow’s Department of Philosophy, has developed the concept of Complexity of Value independent of Less Wrong. 

As far as I can tell Less Wrong does not know of Carter, the only references to his existence I could find on LW and OB were written by me.  Whether Carter knows of LW or OB is harder to tell, but the only possible link I could find online was that he has criticized the views of Michael Huemer, who knows Bryan Caplan, who knows Robin Hanson. This makes it all the more interesting that Carter has developed views on value and morality very similar to ones commonly espoused on Less Wrong.

The Complexity of Value is one of the more important concepts in Less Wrong.  It has been elaborated on its wiki page, as well as some classic posts by Eliezer.  Carter has developed the same concept in numerous papers, although he usually refers to it as “a plurality of values” or “multidimensional axiology of value.”  I will focus the discussion on working papers Carter has on the University of Glasgow’s website, as they can be linked to directly without having to deal with a pay wall.  In particular I will focus on his paper "A Plurality of Values."

Carter begins the paper by arguing:

Wouldn’t it be nice if we were to discover that the physical universe was reducible to only one kind of fundamental entity? ... Wouldn’t it be nice, too, if we were to discover that the moral universe was reducible to only one kind of valuable entity—or one core value, for short? And wouldn’t it be nice if we discovered that all moral injunctions could be derived from one simple principle concerning the one core value, with the simplest and most natural thought being that we should maximize it? There would be an elegance, simplicity and tremendous justificatory power displayed by the normative theory that incorporated the one simple principle. The answers to all moral questions would, in theory at least, be both determinate and determinable. It is hardly surprising, therefore, that many moral philosophers should prefer to identify, and have thus sought, the one simple principle that would, hopefully, ground morality.

And it is hardly surprising that many moral philosophers, in seeking the one simple principle, should have presumed, explicitly or tacitly, that morality must ultimately be grounded upon the maximization of a solitary core value, such as quantity of happiness or equality, say. Now, the assumption—what I shall call the presumption of value-monism—that here is to be identified a single core axiological value that will ultimately ground all of our correct moral decisions has played a critical role in the development of ethical theory, for it clearly affects our responses to certain thought-experiments, and, in particular, our responses concerning how our normative theories should be revised or concerning which ones ought to be rejected.

Most members of this community will immediately recognize the similarities between these paragraphs and Eliezer’s essay “Fake Utility Functions.”  The presumption of value monism sounds quite similar to Eliezer’s description of “someone who has discovered the One Great Moral Principle, of which all other values are a mere derivative consequence.”  Carter's opinion of such people is quite similar to Eliezer's. 

While Eliezer discovered the existence of the Complexity of Value by working on Friendly AI, Carter discovered it by studying some of the thornier problems in ethics, such as the Mere Addition Paradox and what Carter calls the Problem of the Ecstatic Psychopath.  Many Less Wrong readers will be familiar with these problems; they have been discussed numerous times in the community.

For those who aren’t, in brief the Mere Addition Paradox states that if one sets maximizing total wellbeing as the standard of value then one is led to what is commonly called the Repugnant Conclusion, the belief that a huge population of people with lives barely worth living is better than a somewhat smaller population of people with extremely worthwhile lives.  The Problem of the Ecstatic Psychopath is the inverse of this, it states that, if one takes average levels of well-being as the standard of value, that a population of one immortal ecstatic psychopath with a nonsentient machine to care for all their needs is better than a population of trillions of very happy and satisfied, but not ecstatic people.

Carter describes both of these problems in his paper and draws an insightful conclusion:

In short, surely the most plausible reason for the counter-intuitive nature of any mooted moral requirement to bring about, directly or indirectly, the world of the ecstatic psychopath is that either a large total quantity of happiness or a large number of worthwhile lives is of value; and surely the most plausible reason for the counter-intuitive nature of any mooted injunction to bring about, directly or indirectly, the world of the Repugnant Conclusion is that a high level of average happiness is also of value.

How is it that we fail to notice something so obvious? I submit: because we are inclined to dismiss summarily any value that fails to satisfy our desire for the one core value—in other words, because of the presumption of value-monism.

Once Carter has established the faults of value monism he introduces value pluralism to replace it.¹  He introduces two values to start with, “number of worthwhile lives” and “the level of average happiness,” which both contribute to “overall value.”  However,  their contributions have diminishing returns,² so a large population with low average happiness and a tiny population with extremely high average happiness are both  worse than a moderately sized population with moderately high average happiness. 

This is a fairly unique use of the idea of the complexity of value, as far as I know.  I’ve read a great deal of Less Wrong’s discussion of the Mere Addition Paradox, and most attempts to resolve it have consisted of either trying to reformulate Average Utilitarianism so that it does not lead to the Problem of the Ecstatic Psychopath, or redefining what "a life barely worth living" means upwards so that it is much less horrible than one would initially think.  The idea of agreeing that increasing total wellbeing is important, but not the be all and end all of morality, did not seem to come up, although if it did and I missed it I'd be very happy if someone posted a link to that thread.

Carter’s resolution of the Mere Addition Paradox makes a great deal of sense, as it manages to avoid every single repugnant and counterintuitive conclusion that Total and Average Utilitarianism draw by themselves while still being completely logically consistent.  In fact, I think that most people who reject the Repugnant Conclusion will realize that this was their True Rejection all along.  I am tempted to say that Carter has discovered Theory X, the hypothetical theory of population ethics Derek Parfit believed could accurately describe the ethics of creating more people without implying any horrifying conclusions.

Carter does not stop there, however, he then moves to the problem of what he calls “pleasure wizards” (many readers may be more familiar with the term “utility monster”).  The pleasure wizard can convert resources into utility much more efficiently than a normal person, and hence it can be argued that it deserves more resources.  Carter points out that:

…such pleasure-wizards, to put it bluntly, do not exist... But their opposites do. And the opposites of pleasure-wizards—namely, those who are unusually inefficient at converting resources into happiness—suffice to ruin the utilitarian’s egalitarian pretensions. Consider, for example, those who suffer from, what are currently, incurable diseases. … an increase in their happiness would require that a huge proportion of society’s resources be diverted towards finding a cure for their rare condition. Any attempt at a genuine equality of happiness would drag everyone down to the level of these unfortunates. Thus, the total amount of happiness is maximized by diverting resources away from those who are unusually inefficient at converting resources into happiness. In other words, if the goal is, solely, to maximize the total amount of happiness, then giving anything at all to such people and spending anything on cures for their illnesses is a waste of valuable resources. Hence, given the actual existence of such unfortunates, the maximization of happiness requires a considerable inequality in its distribution.

Carter argues that, while most people don’t think all of society’s resources should be diverted to help the very ill, the idea that they should not be helped at all also seems wrong.  He also points out that to a true utilitarian the nonexistence of pleasure wizards should be a tragedy:

So, the consistent utilitarian should greatly regret the non-existence of pleasure-wizards; and the utilitarian should do so even when the existence of extreme pleasure-wizards would morally require everyone else to be no more than barely happy.

Yet, this is not how utilitarians behave, he argues, rather:

As I have yet to meet a utilitarian, and certainly not a monistic one, who admits to thinking that the world would be a better place if it contained an extreme pleasure-wizard living alongside a very large population all at that level of happiness where their lives were just barely worth living…But if they do not  bemoan the lack of pleasure-wizards, then they must surely value equality directly, even if they hide that fact from themselves. And this suggests that the smile of contentment on the faces of utilitarians after they have deployed diminishing marginal utility in an attempt to show that their normative theory is not incompatible with egalitarianism has more to do with their valuing of equality than they are prepared to admit.

Carter resolves the problem of "pleasure wizard" by suggesting equality as an end in itself as a third contributing value towards overall value.  Pleasure wizards should not get all the resources because equality is valuable for its own sake, not just because of diminishing marginal utility.  As with average happiness and total worthwhile lives, equality is balanced against other values, rather than dominating them.   It may often be ethical for a society to sacrifice some amount of equality to increase the total and average wellbeing. 

Carter then briefly states that, though he only discusses three in this paper, there are many other dimensions of value that could be added.  It might even be possible to add some form of deontological rules or virtue ethics to the complexity of value, although  they would be traded off against consequentialist considerations.  He concludes the paper by reiterating that:

Thus, in avoiding the Repugnant Conclusion, the Problem of the Ecstatic Psychopath and the problems posed by pleasure-wizards, as well as the problems posed by any unmitigated demand to level down, we appear to have identified an axiology that is far more consistent with our considered moral judgments than any entailing these counter-intuitive implications.

Carter has numerous other papers discussing the concept in more detail, but “A Plurality of Values” is the most thorough.  Other good ones include “How to solve two addition paradoxes and avoid the Repugnant Conclusion,” which more directly engages the Mere Addition Paradox and some of its defenders like Michael Huemer; "Scrooge and the Pleasure Witch," which discusses pleasure wizards and equality in more detail; and “A pre-emptive response to some possible objections to a multidimensional axiology with variable contributory values,” which is exactly what it says on the tin.

On closer inspection it was not hard to see why Carter had developed theories so close to those of Eliezer and other members of Less Wrong and SIAI communities.   In many ways their two tasks are similar. Eliezer and the SIAI are trying to devise a theory of general ethics that cannot be twisted into something horrible by a rules-lawyering Unfriendly AI, while Carter is trying to devise a theory of population ethics that cannot be twisted into something horrible by rules-lawyering humans.  The worlds of the Repugnant Conclusion and the Ecstatic Psychopath are just the sort of places a poorly programmed AI with artificially simple values would create.

I was very pleased to see an important Less Wrong concept had a defender in mainstream academia.  I was also pleased to see that Carter had not just been content to develop the concept of the Complexity of Value.    He was also able to employ in the concept in new way, successfully resolving one of the major quandaries of modern philosophy.

Footnotes

¹I do not mean to imply Carter developed this theory out of thin air of course. Value pluralism has had many prominent advocates over the years, such as Isaiah Berlin and Judith Jarvis Thomson.

²Theodore Sider proposed a theory called "geometrism" in 1991 that also focused on diminishing returns, but geometrism is still a monist theory, it had geometric diminishing returns for the people in the scenario, rather than the values creating the people was trying to fulfill.

Edited - To remove a reference to Aumann's Agreement Theorem that the commenters convinced me was unnecessary and inaccurate.

Cross-posted from http://www.robertwiblin.com

There is a principle in finance that obvious and guaranteed ways to make a lot of money, so called ‘arbitrages’, should not exist. It has a simple rationale. If market prices made it possible to trade assets around and in the process make a guaranteed profit, people would do it, in so doing shifting some prices up and others down. They would only stop making these trades once the prices had adjusted and the opportunity to make money had disappeared. While opportunities to make ‘free money’ appear all the time, they are quickly noticed and the behaviour of traders eliminates them. The logic of selfishness and competition mean the only remaining ways to make big money should involve risk taking, luck and hard work. This is the ’no arbitrage‘ principle.

Should a similar principle exist for selfless as well as selfish finance? When a guaranteed opportunity to do a lot of good for the world appears, philanthropists should notice and pounce on it, and only stop shifting resources into that activity once the opportunity has been exhausted. This wouldn’t work as quickly as the elimination of arbitrage on financial markets of course. Rather it would look more like entrepreneurs searching for and exploiting opportunities to open new and profitable businesses. Still, in general competition to do good should make it challenging for an altruistic start-up or budding young philanthropist to beat existing charities at their own game.

There is a very important difference though. Most investors are looking to make money and so for them a dollar is a dollar, whatever business activity it comes from. Competition between investors makes opportunities to get those dollars hard to find. The same is not true of altruists, who have very diverse preferences about who is most deserving of help and how we should help them; a ‘util’ from one charitable activity is not the same as a ‘util’ from another. This suggests that unlike in finance, we may able to find ‘altruistic arbitrages’, that is to say ‘opportunities to do a lot of good for the world that others have left unexploited.’

The rule is simple: target groups you care about that other people mostly don’t, and take advantage of strategies other people are biased against using.  That rule is the root of a lot of advice offered to thoughtful givers and consequentialist-oriented folks. An obvious example is that you shouldn’t look to help poor people in rich countries. There are already a lot of government and private dollars chasing opportunities to assist them, so the low hanging fruit has all been used up and then some. The better value opportunities are going to be in poor, unromantic places you have never heard of, where fewer competing philanthropist dollars are directed. Similarly, you should think about taking high risk-high return strategies. Most do-gooders are searching for guaranteed and respectable opportunities to do a bit of good, rather than peculiar long-shot opportunities to do a lot of good. If you only care about the ‘expected‘ return to your charity, then you can do more by taking advantage of the quirky, improbable bets neglected by others.

Who do I personally care about more than others? For me the main candidates are animals, especially wild ones, and people who don’t yet exist and may never exist – interest groups that go largely ignored by the majority of humanity. What are the risky strategies I can employ to help these groups? Working on future technologies most people think are farcical naturally jumps to mind but I’m sure there are others and would love to hear them.

This principle is the main reason I am skeptical of mainstream political activism as a way to improve the world. If you are part of a significant worldwide movement, it’s unlikely that you’re working in a neglected area and exploiting how your altruistic preferences are distinct from those of others.

What other conclusions can we draw thinking about philanthropy in this way?

"Vulnerable Cyborgs: Learning to Live with our Dragons", Mark Coeckelbergh (university); abstract:

Transhumanist visions appear to aim at invulnerability. We are invited to fight the dragon of death and disease, to shed our old, human bodies, and to live on as invulnerable minds or cyborgs. This paper argues that even if we managed to enhance humans in one of these ways, we would remain highly vulnerable entities given the fundamentally relational and dependent nature of posthuman existence. After discussing the need for minds to be embodied, the issue of disease and death in the infosphere, and problems of psychological, social and axiological vulnerability, I conclude that transhumanist human enhancement would not erase our current vulnerabilities, but instead transform them. Although the struggle against vulnerability is typically human and would probably continue to mark posthumans, we had better recognize that we can never win that fight and that the many dragons that threaten us are part of us. As vulnerable humans and posthumans, we are at once the hero and the dragon.

Bostrom has written a tale about a dragon that terrorizes a kingdom and people who submit to the dragon rather than fighting it. According to Bostrom, the “moral” of the story is that we should fight the dragon, that is, extend the (healthy) human life span and not accept aging as a fact of life (Bostrom 2005, 277). And in The Singularity is Near (2005) Kurzweil has suggested that following the acceleration of information technology, we will become cyborgs, upload ourselves, have nanobots in our bloodstream, and enjoy nonbiological experience. Although not all transhumanist authors explicitly state it, these ideas seem to aim toward invulnerability and immortality: by means of human enhancement technologies, we can transcend our present limited existence and become strong, invulnerable cyborgs or immortal minds living in an eternal, virtual world.

...However, in this paper, I will ask neither the ethical-normative question (Should we develop human enhancement techniques and should we aim for invulnerability?) nor the hermeneutical question (How can we best interpret and understand transhumanism in the light of cultural, religious, and scientific history?). Instead, I ask the question: If and to the extent that transhumanism aims at invulnerability, can it – in principle – reach that aim? The following discussion offers some obvious and some much less obvious reasons why posthumans would remain vulnerable, and why human vulnerability would be transformed rather than diminished or eliminated...However, to focus only on a defense or rejection of what is valuable in humans would leave out of sight the relation between (in)vulnerability and posthuman possibilities. It would lead us back to the ethical-normative questions (Is human enhancement morally acceptable? Is vulnerability something to be valued? Is the transhumanist project acceptable or desirable?), which is not what I want to do in this paper. Moreover, ethical arguments that present the problem as if we have a choice between “natural” humanity and “artificial” posthumanity are based on essentialist assumptions that make a sharp distinction between “what we are” (the natural) and technology (the artificial), whereas this distinction is at least questionable. Perhaps there is no fixed human nature apart from technology, perhaps we are “artificial by nature” (Plessner 1975). If this is so, then the problem is not whether or not we want to transcend the human but how we want to shape that posthuman existence. Should we aim at invulnerability and if so, can we? As indicated before, here I limit the discussion to the “can” question.

Breaking down the potential improvements:

Physical vulnerability

Not only could human enhancement make us immune to current viruses; it could also offer other “immunities,” broadly understood...However, the project of total vulnerability or even overall reduction of vulnerability is bound to fail. If we consider the history of medical technology, we observe that for every disease new technology helps to prevent or cure, there is at least one new disease that escapes our techno-scientific control. We can win one battle, but we can never win the war. There will be always new diseases, new viruses, and, more generally, new threats to physical vulnerability. Consider also natural disasters caused by floods, earthquakes, volcanic eruptions, and so on.

Moreover, the very means to fight those threats sometimes create new threats themselves. This can happen within the same domain, as is the case with antibiotics that lead to the development of more resistant bacteria, or in another domain, as is the case with new security measures in airports, which are meant as protections against physical harm by terrorism but might pose new (health?) risks. Paradoxically, technologies that are meant to reduce vulnerability often create new ones. This is also true for posthuman technologies. For example, posthumans would also be vulnerable to at least some of the risks Bostrom calls “existential risks” (Bostrom 2002), which could wipe out posthumankind. Nanotechnology or nuclear technology could be misused, a superintelligence could take over and annihilate humankind, or technology could cause (further) resource depletion and ecological destruction. Military technologies are meant to protect us but they can become a threat, making us vulnerable in a new way. We wanted to master nature in order to become less dependent on it, but now we risk destroying the ecology that sustains us. And of course there are many physical threats we cannot foresee – not even in the near future.

Material and immaterial vulnerability

Consider computer viruses. Here the story is similar to the story of biological viruses: there are ongoing cycles of threats, counter-measures, and new threats. We can also consider physical damage to computers, although that is much less common. In any case, if we extend ourselves with software and hardware, this creates additional vulnerabilities. We must cope with “software” vulnerability and “hardware” vulnerability. If humans and posthumans live in an “infosphere” (see for example Floridi 2002), this is not a sphere of immunity. Perhaps our vulnerability becomes less material, but we cannot escape it. For instance, a virtual body in a virtual world may well be shielded from biological viruses, but it is vulnerable to at least three kinds of threats.

1. First, there are threats within the virtual world itself (consider for instance virtual rape), which constitutes virtual vulnerability.
   
2. Second, the software programme that provides a platform for the virtual world might be damaged, for example by means of a cyber attack. This can lead to the “death” of the virtual character or entity.
   
3. Third, all these processes depend on (material) hardware. The world wide web and its wired and wireless communications rest on material infrastructures without which the web would be impossible. Therefore, if posthumans uploaded themselves into an infosphere and dispensed with their biological bodies, they would not gain invulnerability and immortality but merely transform their vulnerability.

Bodily vulnerability

Minds need bodies. This is in line with contemporary research in cognitive science, which argues that “embodiment” is necessary since minds can develop and function only in interaction with their environment (Lakoff and Johnson 1999 and others). This direction of thought is also taken in contemporary robotics, for example when it recognizes that manipulation plays an important role in the development of cognition (Sandini et al. 2004). In his famous 1988 book on “mind children” Moravec argued that true AI can be achieved only if machines have a body (Moravec 1988)...Thus, uploading and nano-based cyborgization would not dispense with the body but transform it into a virtual body or a nano-body. This would create vulnerabilities that sometimes resemble the vulnerabilities we know today (for instance virtual violence) but also new vulnerabilities.

Metaphysical vulnerability

With this atomism comes that atomist view of death: there is always the possibility of disintegration; neither physical-material objects nor information objects exist forever. Information can disintegrate and the material conditions for information are vulnerable to disintegration as well. Thus, at a fundamental level everything is vulnerable to disintegration, understood by atomism as a re-organization of elementary particles. This “metaphysical” vulnerability is unavoidable for posthumans, whatever the status of their elementary particles and the organs and systems constituted by these particles (biological or not). According to their own metaphysics, the cyborgs and inforgs that transhumanists and their supporters wish to create would be only temporal orders that have only temporary stability – if any.

 Note, however, that recently both Floridi and contemporary physics seem to move toward a more ecological, holistic metaphysics, which suggests a different definition of death. In information ecologies, perhaps death means the absence of relations, disconnection. Or it means: deletion, understood ecologically and holistically as the removal out of the whole. But in the light of this metaphysics, too, there seems no reason why posthumans would be able to escape death in this sense.

Existential and psychological vulnerabilities

This gives rise to what we may call “indirect” or “second-order” vulnerabilities. For instance, we can become aware of the possibility of disintegration, the possibility of death. We can also become aware of less threatening risks, such as disease. There are many first-order vulnerabilities. Awareness of them renders us extra vulnerable as opposed to beings who lack such an ability to take distance from ourselves. From an existential-phenomenological point of view (which has its roots in work by Heidegger and others), but also from the point of view of common sense psychology, we must extend the meaning of vulnerability to the sufferings of the mind. Vulnerability awareness itself constitutes a higher-order vulnerability that is typical of humans. In posthumans, we could only erase this vulnerability if we were prepared to abandon the particular higher form of consciousness that we “enjoy.” No transhumanist would seriously consider that solution to the problem.

Social and emotional vulnerability

If I depend on you socially and emotionally, then I am vulnerable to what you say or do. Unless posthumans were to live in complete isolation without any possibility of inter-posthuman communication, they would be as vulnerable as we are to the sufferings created by the social life, although the precise relation between their social life and their emotional make-up might differ...For example, in Houellebecq’s novel the posthumans have a reduced capacity to feel sad, but at the cost of a reduced capacity to desire and to feel joy. More generally, the lesson seems to be: emotional enhancement comes at a high price. Are we prepared to pay it? Even if we succeed in diminishing this kind of vulnerability, we might lose something that is of value to us. This brings me to the next kind of vulnerability.

Ethical-axiological vulnerability

We value not only people and our relationships with them; we are also attached to many other things in life. Caring makes us vulnerable (Nussbaum 1986). We develop ties out of our engagement with humans, animals, objects, buildings, landscapes, and many other things. This renders us vulnerable since it makes us dependent on (what we experience as) “external” things. We sometimes get emotional about things since we care and since we value. We suffer since we depend on external things...Posthumans could be cognitively equipped to follow this strategy, for instance by means of emotional enhancement that allows more self-control and prevents them forming too strong ties to things. If we really wanted to become invulnerable in this respect, we should create posthumans who no longer care at all about external things – including other posthumans. That would be “posthumans” who no longer have the ability to care and to value. They would “connect” to others and to things, but they would not really engage with them, since that would render them vulnerable. They would be perfectly rational Stoics, perhaps, but it would be odd to call them “posthumans” at all since the term “human” would lose its meaning. It is even doubtful if this extreme form of Stoicism would be possible for any entity that possesses the capacity of valuing and that engages with the world.

'Relational vulnerability'/'Conclusion: Heels and dragons'

The only way to make an entity invulnerable, it turns out, would be to create one that exists in absolute isolation and is absolutely independent of anything else. Such a being seems inconceivable – or would be a particularly strange kind of god. (It would have to be a “philosopher’s” god that could hardly stir any religious feelings. Moreover, the god would not even be a “first mover,” let alone a creator, since that would imply a relation to our world. It is also hard to see how we would be aware of its existence or be able to form an idea about it, given the absence of any relation between us and the god.) Of course we could – if ethically acceptable at all – create posthumans that are less vulnerable in some particular areas, as long as we keep in mind that there are other sources of vulnerability, that new sources of vulnerability will emerge, and that our measure to decrease vulnerability in one area may increase it in another area.

If transhumanists accept the results of this discussion, they should carefully reflect on, and redefine, the aims of human enhancement and avoid confusion about how these aims relate to vulnerability. If the aim is invulnerability, then I have offered some reasons why this aim is problematic. If their project has nothing to do with trying to reach invulnerability, then why should we transcend the human? Of course one could formulate no “ultimate” goals and choose less ambitious goals, such as more health and less suffering. For instance, one could use a utilitarian argument and say that we should avoid overall suffering and pain. Harris seems to have taken these routes (Harris 2007). And Bostrom frequently mentions “life extension” as a goal rather than “invulnerability” or “immortality.” But even in these “weakened” or at least more modest forms, the transhumanist project can be interpreted as a particularly hostile response to (human) vulnerability that probably has no parallel in human history.

...Furthermore, this paper suggests that if we can and must make an ethical choice at all, then it is not a choice between vulnerable humans and invulnerable posthumans, or even between vulnerability and invulnerability, but a choice between different forms of humanity and vulnerability. If implemented, human enhancement technologies such as mind uploading will not cancel vulnerability but transform it. As far as ethics is concerned, then, what we need to ask is which new forms of the human we want and how (in)vulnerable we wish to be. But this inquiry is possible only if we first fine-tune our ideas of what is possible in terms of enhancement and (in)vulnerability. To do this requires stretching our moral and technological imaginations.

Moreover, if I’m right about the different forms of posthuman vulnerability as discussed above, then we must dispense with the dragon metaphor used by Bostrom: vulnerability is not a matter of “external” dangers that threaten or tyrannize us, but that have nothing to do with what we are; instead, it is bound up with our relational, technological and transient kind of being – human or posthuman. If there are dragons, they are part of us. It is our tragic condition that as relational entities we are at once the heel and the arrow, the hero and the dragon.

Before criticizing it, I'd like to point to the introduction where the author lays out his mission: to discuss what problems cannot "in principle" be avoided, what vulnerabilities are "necessary". In other words, he thinks he is laying out fundamental limits, on some level as inexorable and universal as, say, Turing's Halting Theorem.

But he is manifestly doing no such thing! He lists countless 'vulnerabilities' which could easily be circumvented to arbitrary degrees. For example, the computer viruses he puts such stock on: there is no fundamental reason computer viruses must exist. There are many ways they could be eliminated starting from formal static proofs of security and functionality; the only fundamental limit relevant here would be Turing/Rice's theorem, which is applicable only if we wanted to run all possible programs, which we manifestly cannot and do not. Similar points apply to the rest of his software vulnerabilities.

I would also like to single out his 'Metaphysical vulnerability'; physicists, SF authors, and transhumanists have been, for decades, outlining a multitude of models and possibilities for true immortality, ranging from Dyson's eternal intelligences to Tipler's collapse to Omega point to baby blackhole-universes. To appeal to atomism is to already beg the question (why not run intelligence on waves or more exotic forms of existence, why this particle-chauvinism?).

This applies again and again - the author supplies no solid proofs from any field, and apparently lacks the imagination or background to imagine ways to circumvent or dissolve his suggested limits. They may be exotic methods, but they still exist; were the author to reply that to employ such methods would result in intelligences so alien as to no longer be human, then I should accuse him of begging the question on a even larger scale - of defining the human as desirable and, essentially, as that which is compatible with his chosen limits.

Since that question is at the heart of transhumanism, his paper offers nothing of interest to us.

GiveWell, LessWrong's most cited organization for optimal philanthropy, is currently re-evaluating its charity rankings with the goal of naming a new #1 charity by December 2011. Essentially, VillageReach (the current top charity) has met all of its short-term funding needs, to the point where it no longer has the greatest marginal return.

Our current top-rated charity is VillageReach. In 2010, we directed over $1.1 million to it, which met its short-term funding needs (i.e., its needs for the next year or so).

VillageReach still has longer-term needs, and in the absence of other giving opportunities that we consider comparable, we’ve continued to feature it as #1 on our website. However, we’ve also been focusing most of our effort this year on identifying and investigating other potential top-rated charities, with the hope that we can refocus attention on an organization with shorter-term needs this December. (In general, the vast bulk of our impact on donations comes in December.) We believe that we will be able to do so. We don’t believe we’ll be able to recommend a giving opportunity as good as giving to VillageReach was last year, but given VillageReach’s lack of short-term (1-year) room for more funding, we do expect to have a different top recommendation by this December.

EDIT: The new charities are up! They are the Against Malaria Foundation and the Schistosomiasis Control Initiative.

Utilitarianism seems to be a common theme on this site. I suggest checking out felicifia.org, a Utilitarianism forum. That is all.

I've begun an online discussion with Alan Dawrst (Brian Tomasik) of utilitarian-essays.com concerning Friendly AI and utilitarianism. Interested parties may wish to follow along or participate.

The forum thread now contains many overlapping discussions. For clarity, here's an index of the narrow, core discussion between Alan and I:

1. Luke #1
2. Alan #1
3. Luke #2
4. Alan #2
5. Luke #3
6. Alan #3
7. Luke #4
8. Alan #4
9. Luke #5
10. Alan #5
11. Luke #6
12. Alan #6
13. Luke #7
14. Alan #7

There are a lot of explanations of consequentialism and utilitarianism out there, but not a lot of persuasive essays trying to convert people. I would like to fill that gap with a pro-consequentialist FAQ. The target audience is people who are intelligent but may not have a strong philosophy background or have thought about this matter too much before (ie it's not intended to solve every single problem or be up to the usual standards of discussion on LW).

I have a draft up at http://www.raikoth.net/consequentialism.html (yes, I have since realized the background is horrible, and changing it is on my list of things to do). Feedback would be appreciated, especially from non-consequentialists and non-philosophers since they're the target audience.

"Wet houses"-- subsidized housing for alcoholics (they need to get most of their own money for alcohol, but their other expenses are covered) might actually be a good idea. It's cheaper than trying to get them to stop drinking, arguably kinder than trying to get people to take on a very hard task that they aren't interested in, and leads to less collateral damage than having alcoholics couch-surfing or living on the street.

Utilitarians, what do you think?

From Mike Darwin's new blog:

DCD has lead to a fracture within the medical community [7,8] wherein some centers, such as the University of Pittsburgh Medical Center, have taken patients who want ventilator support withdrawn, placed femoral cannulae under local (spinal) anesthesia, turned off the ventilator after effectively anesthetizing the patient, waited until the patient’s heart stops, and then restarted circulation with CPB. They also, of course, give paralytic neuromuscular blocking drugs (as is routine in all visceral organ retrieval) to prevent the thoracoabdominal incision, and the terminal drop in blood pressure (when the organs are removed), from causing muscle vesiculations (twitching) or actual limb movement as a result of stimulation of the nocioceptive pathways in the spinal cord (pain is a local phenomenon first and a central nervous system one secondly with the process proceeding up the spinal cord to the brain). [9,10]

To be blunt, this procedure resulted in all hell breaking out. [11,12,13] Bioethicists, such James Bernat and Leslie Whetstine, accused the surgeons and neurologists involved in this undertaking of every ethical evil, including homicide.[14,15] A compromise position is to restore circulation in the body using a special balloon-tipped aortic catheter that prevents ‘all ‘ flow to the brain. This results in a ‘resolution’ to the ‘paradox’ of removing organs from a patient with a ‘viable, or potentially viable brain.’ Of course, from our perspective as cryonicists, this whole exercise is nothing more or less than a procedural contortion designed to avoid confronting the reality that death is not a binary condition, and that if you are going to allow people to withdraw from medical care they no longer want, and that they (rightfully) consider an assault, then the corollary to that is that they also get to decide when they are dead. [16] That means that they have the perfect right to ask for, and receive a treatment (i.e., in the presence of informed consent) whereby they are anesthetized, cooled, subjected to blood washout, and their organs removed – at which point they are indeed DEAD, in the sense that their non-functional condition is now irreversible, or not going to be reversed, because they do not want it to be. When, exactly, they become irrecoverable from an information-theoretic standpoint is irrelevant, because they don’t want to be recovered, and no technology currently exists that will allow them to be recovered.

We, as cryonicists, could argue that if such patients were cryopreserved, they might possibly be recovered in the future. But if they do not want cryopreservation, then they are dead when they say they are dead, and when they meet the current medico-legal definition of cardiorespiratory death (i.e., no heartbeat or breathing and no prospect of their resuming). The medical response to this fairly straightforward situation has been, as expected, convoluted and irrational, and profoundly dangerous to cryonics. The recent paper “Clarifying the paradigm for the ethics of donation and transplantation: Was ‘dead’ really so clear before organ donation?” [17] is an excellent window into current medical policy, not just on the issue of DCD, but on the application of any kind of circulatory support to patients who have been pronounced dead on the basis of clinical (cardiac) criteria.  This article is one of the most cited in current DCD debates, and the closing sentence in its abstract says it all (emphasis mine):

“Criticism of controlled DCD on the basis of violating the dead donor rule, where autoresuscitation has not been described beyond 2 minutes, in which life support is withdrawn and CPR is not provided, is not valid. However, any post mortem intervention that reestablishes brain blood flow should be prohibited. “In comparison to traditional practice, organ donation has forced the clarification of the diagnostic criteria for death and improved the rigour of the determinations.”[17]

...

The UK has already adopted standards for determining and pronouncing death that expressly prohibit the application of CPR, or any modalities that restore flow to the brain or conserve brain viability. I have made inquiries, and been informed that failure to follow these Guidelines would be a serious breach of professional conduct, resulting in any licensed person being struck off; and that such action would very likely constitute a criminal act in the UK, as well (prosecution to be at the discretion of law enforcement and the prosecutor). [21]

The whole point of cryonics -- not to put too fine a point on it -- is to conserve brain viability, in the sense of keeping as much of the brain in as close to a viable state as possible.

ETA: Mike has confirmed that the UK law applies to non organ donors. He also has stated that new changes have been made to the Uniform Anatomical Gift Act (a sort of template by which state laws are drafted) which are likely to be similar in nature, in the US.

A distinction that some people grok right away and some others may not realize exists:

Imagine a country called “Lanmindia,” where much of the population has seen its legs blown off in horrible accidents.  Does that sound like a pretty miserable place?  Happiness research suggests not.  The claim is that there is a sort of natural “set-point” for happiness, and that after winning a lottery one is happy for a short time, and then you revert right back to your natural happiness level.  I find that plausible.  They also claim that if someone loses a limb, then they are unhappy for a short period and then revert back to normal.  I find that implausible, but if the evidence says it is the case then I guess I need to accept that.

My claim is that although Lanmindia is just as happy as America, it has much lower utility.  Let’s define ’utility’ as ”that which people maximize.”  People very much don’t want to have their legs blown off, and hence emigrate from Lanmindia in droves.  People behave as if they care about utility, not happiness.

-Scott Sumner, "Nonsense on stilts: Part 1.  What if utility and happiness are unrelated?" TheMoneyIllusion

This is also somewhat a reply to Hanson's "Lift Up Your Eyes" on Overcoming Bias. Some people on LessWrong are careful to make the distinction between ordinal utility, cardinal utility, and fuzzies, and others aren't quite so much. The above sentence on accepting evidence and the post script that he is not serious about one part of the post might also make interesting conversation -- part two is advice to move next door to a child molester for cheaper housing if you don't have a kid and part three is about The Fed taking advantage of banks.

It has been suggested that animals have less subjective experience than people. For example, it would be possible to have an animal that counts as half a human for the purposes of morality. This is an argument as to why that may be the case.

If you're moving away from Earth at 87% of the speed of light, time dilation would make it look like time on Earth is passing half as fast. From your point of reference, everyone will live twice as long. This obviously won't change the number of life years they live. You can't double the amount of good in the world just by moving at 87% the speed of light. It's possible that there's just a preferred point of reference, and everything is based on people's speed relative to that, but I doubt it.

No consider if their brains were slowed down a different way. Suppose you uploaded someone, and made the simulation run at half speed. Would they experience a life twice as long? This seems to be just slowing it down a different way. I doubt it would change the total amount experienced.

If that's true, it means that sentience isn't something you either have or don't have. There can be varying amounts of it. Also, someone whose brain has been slowed down would be less intelligent by most measures, so this is some evidence that subjective experience correlates with intelligence.

Edit: replaced "sentience" with the more accurate "subjective experience".