Less Wrongers are a diverse crowd, more so now than in the early days.  I wonder if we could step away from anti-generalizations, generalize and try to say good places to live, under a few assumptions (remember, the idea of an assumption is to assume it, not to claim it is less or more representative of observation class X or Y and then go on to nerdify it.)

Recetly, Xanghai was claimed as an interesting place to teach english. 

Just having returned from 15 days in Rio de Janeiro, I may talk a little about it.

Assumptions:

1) Assuming your family lives somewhere else, other state or country.

2) No children yet. Single, Married, Gay, Bisexual, Male, Female.

3) You can muster $1-4k a month (teaching a language, like English, programming, writing, family money, lottery, spy for the CIA)

4) You like science/philosophy, rationality, and not a complete misanthrope (you'd hug five times more than you do if given a chance, and you'd double the number of close friends you have, as well as balance their gender ratio)

 

My suggested format is city name, time spend there, experience, cons, and pros.

Rio de Janeiro,15 days, Rio is an interesting city. Near the subway you can get to the vast majority of places without a car, a good night out will cost between 15-40 dollars, depending on whether you drink or not, and therefore need a cab home. Nice dinner 12-50.  There are millions of people including lots of tourists easily reachable there. So unless you are estonian, you will be able to find someone from home there. Because travellers go to Rio for it's beauty, you can find them in free places, and make friends with locals and foreigners alike, allowing for short term and long term friendships. They say you get tired eventually, but the natural beauty is great and spread. Forests and beaches and mountains abound, all 4 minutes away from a supermarket.There are nearly free public bikes in some areas.

Cons: Science/philosophy are not what Rio is known for. Their universities are good, and you can find youe way there if you can in a good college, but a meeting with a lot of people to discuss two boxing on newcomb is less likely in the following ten years.You can't park in Rio during the day, if somehow you managed to have a car and a carplace in your apartment. You won't buy a place,and it won't be big, an awesome ipanema apartment 190sq meters goes for 2,3 million dollars, and renting a tiny place costs about 1thousand a month.

Pros: Papers to the contrary, weather does impact your life for quite a while if you pay attention to it. Not necessarily the weather itself, but the social oppotunities that arise because of it (moonlight music at the beach, free overhearing music in the bohemian neighborhood, dancing as opposed to freezing, etc...) can be, literally, life-changing.  Rio has many people not from Rio, so it is easy to befriend them, they also need new friends.  The Couchsurfing community is active and speaks english.

Neutral: Many think that people (specially women) look amazing in Brazil, quite the contrary. Our average look is way below your expectations, but the top5% of people are really better looking to foreigner eyes than the 5%of their own country. Long tails, pun intended.

If you lived for a while in a city that you'd like to recommend to some niche Less Wrongers, report. Avoid doing so for the city you were born in, since a native experience differs violently from a migrant/immigrant experience.

 

There is a bit of nice and recent discussion going on about money here, "money" in the sense of the common peasant "I want more money" not in the senses of "Unit of caring" "Utilon buyer" "Trade-able entity unavailable for acausal trade" etc...

There is also, on the web, and in your local store, hundreds of thousands of advices on how to make money: fast, more, passively, selling your body, or brain, or virtual structures. 

People who have money (entrepreneurs, owners), study money (economists) or focus on money (stockbrokers) have a lot to tell you about something closely related. Incentives. They studied and understand when incentives work and when they don't. Some have become masters of creating the right incentives, for employees, customers, CEOs, everyone.

I wont lie, usually, the incentive is money. Pure, abstract, trade-able, feeling-of-power causing, money. Sometimes (Citibank, Safra Bank) also travelling.

What I don't see around a lot, and would like to ask you to help me brainstorm, and list (later I'll edit by adding suggestions and maybe commenting) are the incentives to make money themselves that work, and those that don't.

There are a lot of LWers, I know, who excel at domain X and are emotionally averse to making money, out of X, or sometimes, at all. Money-making is, in a word, impure! Item 6 will deal with that.

I'll say some random unconventional things and open the discussion for those with more experience:

1)Belonging to a society that lacks money and money making altogether is a very powerful incentive for wanting money. The Indians of Brazil (my homeland) frequently engage in juridic battles and bureaucracy to obtain more money from governmental organs, more time than they would dedicate to get medical attention, for instance.

2)I've heard that having a lot of money is an incentive to make more, by making it easier to do so. Spiteful Billionaire meme makes fun of this hypothesis

3)Also heard that not having money is a great incentive to make more. This seems more logical. When you see the empty plate in the horizon, or being thrown in the streets, some sort of alarm must trigger. (If you have experience with this, relate it please)

Yet, I find that even though there is a strong correlation (within the brazilian cultural elite I hang around with) between having a job and not having money at ages 18-28, there is no correlation at all between not having money and efficiency of making money (by either already be making a lot, or in a career with such prospect). It is as if the emergency button can get you to hike a hill, but not to climb a mountain.

4)Morals against munching on others. Traditional, moral or religious frameworks of mind will make an individual believe (probably truly) that munching on friends, family, couchsurfers, franciscans, altruists and other exploitable folk is a bad/undesirable thing to do. One who doesn't want to munch has a stronger incentive to get his own money.

5)Pride. Lots of people are very very proud of having money, theirs or familial. I never understood that, but here it is, just a fact, stated.

6)Nobility times. My mum points out frequently that in our day and age (in Brazil, but also in the states) there is an idealization of making money. Sometimes it is great, sometimes it is nauseating. Her point though is that during the medieval ages when social class was fundamentally determined from birth, working was seen as a lower activity. It was nearly opposite of Self-Madesmanship. Only the needy, with little status shall work. Same view was held by Aristotle, who, in his conception of working, never worked a day in his life. Something like how we see manual tough labour nowadays, but looking down on every worker. Some people, her and me included, were raised somehow in this anomic (mislocated) environment which persists in many European colonies and perhaps among the descendants, within Europe, of past nobles. Worse for them in a changing world.

7) Your suggestions will be listed here soon...

I ask you to suggest things which are incentives for, or against making money. And I don't mean "Good Reasons" and "Bad Reasons" I mean which incentives, in economic jargon, work effectively as an incentive for people to make more money. Then after that you can write about their goodness and badness.

 

 

Many of us are familiar with the marshmallow test.If you are not, here.

It is predictive of success, income, level of education, and several other correlated measures.

I'm here to argue for the marshmallow eaters, as a devil's advocate. Contra Ainslie, for instance. I do it out of genuine curiosity, real suspicion, and maybe so that smart people get me back to my original position, pro-long term.

There is also the e-marshmallow test (link is not very relevant), in which children have to face the tough choice between surfing an open connected computer with games, internet etc... and waiting patiently for the experimenter to get back. Upon the experimenter's arrival, they get a pile of marshmallows. I presume it also correlates with interesting things, though haven't found much on it.

I have noticed that rationalists, LessWrongers, Effective Altruists, Singularitarians, Immortalists, X-risk worried folk, transhumanists, are all in favor of taking the long view.  Nick Bostrom starts his TED by saying: "I've been asked to take the long view"

I haven't read most of Less Wrong, but did read the sequences, the 50 top scoring posts and random posts. The overwhelming majority view is that the long view is the most rational view. The long term perspective is the rational way for agents to act.

Lukeprog, for instance, commented:

"[B]ut imagine what one of them could do if such a thing existed: a real agent with the power to reliably do things it believed would fulfill its desires. It could change its diet, work out each morning, and maximize its health and physical attractiveness."

To which I responded:

I fear that in this phrases lies one of the big issues I have with the rationalist people I've met thus far. Why would there be a "one" agent, with "its" desires, that would be fulfilled. Agents are composed of different time-spans. Some time-spans do not desire to diet. Others do (all above some amount of time). Who is to say that the "agent" is the set that would be benefited by those acts, not the set that would be harmed by it.
My view is that picoeconomics is just half the story.
In this video, I talk about picoeconomics from 7:00 to 13:20 I'd suggest to take a look at what I say at 13:20-18:00 and 20:35-23:55, a pyramidal structure of selfs, or agents. 

So you don't have to see the video, let us design a structure of selfhood.

First there is intertemporal conflict, conflict between desires that can be fulfilled at different moments of time. Those reliably fall under a hyperbolic characterization, and the theory that described this is called Picoeconomics, mostly developed by George Ainslie in his Breakdown of Will and elsewhere.

But there is also time-length, or time-span conflict.  The conflict that arises from the fact that you are, at the same time, the entity that will last 200milliseconds, the entity that will last one second, and the entity that will last a year, or maybe, a thousand years.

What do we (humanity) know about personal identity at this point in history? If mainstream anglophone philosophical thought is to be trusted, we have to look for Derek Parfit's work Reasons and Persons, and posterior related work, to get that.

I'll sum it up very briefly: As far as we are concerned, there are facts about continuity of different mental classes. There is continuity of memory, continuity of conscious experience, continuity of psychological traits and tendencies, continuity of character, and continuity of inferential structure (the structure that we use to infer things from beliefs we acquire or access).   

For each of these traits, you can take an individual at two points in time and measure how related I_t1 and I_t2 are with respect to that psychological characteristic.  This is how much I at T₂ is like himself at T₁.

Assign weights for traits according to how much you care (or how important each is in the problem at hand) and you get a composed individual, for which you can do the same exercise, using all of them at once and getting a number between 0 and 1, or a percentage. I'll call this number Self-Relatedness, following the footsteps of David Lewis.

This is our current state of knowledge on Personal Identity: There is Trait-Relatedness, and there is Self-Relatedness. After you know all about those two, there is no extra fact about personal identity. Personal Identity is a confused concept, and when we decompose it into less confused, but more useful, sub-sets, there is nothing left to be the meta-thing "Personal Identity".

Back to the time-length issue, consider how much more me  the shorter term selves are (that is how much more Self-Relatedness there is between any two moments within them). 

Sure if you go all the way down to 10 milliseconds, this stops being true, because there are not even traits to be found. Yet, it seems straightforward that I'm more like me 10 seconds ago than like me 4 months ago, not always, but in the vast majority of cases.

So when we speak of maximizing my utility function, if we overlook what me is made of, we might end up stretching ourselves to as long-term as we possibly can, and letting go of the most instantaneous parts, which de facto are more ourselves than those ones.

One person I met from the LessWrong Singinst cluster claimed: "I see most of my expected utility after the singularity, thus I spend my willpower entirely in increasing the likelihood of a positive singularity, and care little about my current pre-singularity emotions"

Is this an amazing feat of self-control, a proof that we can hope to live according to ideal utility functions after all? Or is it a defunct conception of what a Self is?

I'm not here to suggest a canonical curve of time-lengths of which the Self is composed. Different people are different in this regard. Some time-lengths are stretchable, some can be shortened. Different people will also value the time-lengths differently. 

It would be unreasonable for me to expect that people would, from now on, put on a disclaimer on their writings "I'm assuming 'rational' to mean 'rational to time-lenghts above the X treshold' for this writing". It does, however, seem reasonable to keep an internal reminder when we reason about life choices, decisions, and writings, that not only there are the selves which are praised by the Rationalist cluster, the long term ones, but also, the short term ones.

A decision to eat the marshmallow can, after all, be described as a rational decision, it all depends on how you frame the agent, the child.

So when a superintelligence arises that, despite being Friendly and having the correct goals, does the AGI equivalent of scrolling 9gag, eating Pringles and drinking booze all day long, tell the programmers that the concept of Self, Personal Identity, Agent, or Me-ness was not sufficiently well described, and vit cares too much for vits short-term selves. If they tell you: "Too late, vit is a Singleton already" you just say "Don't worry, just make sure the change is ve-e-e-ery slow..."

How was it?  Which speakers delivered according to expectations?

Which topics were left unresolved?

Were any topics resolved?

Whatever you have to say about it, say it here.

Suggestion: if you are going to comment, mention "I was there" just so we know who was or wasn't.

For very long I've been caring a lot for the preferences of my past selves. 

Rules I established in childhood became sacred, much like laws are (can't find post in the sequences in which Yudkowsky is amazed by the fact that some things are good just because they are old), and that caused interesting unusual life choices, such as not wearing formal shoes and suits. 

I was spending more and more time doing what my previous selves thought I should, in a sense, I was composed mostly of something akin to what Anna Salomon and Steve Rayhawk called Cached Selves.

That meant more dedication to long term issues (Longevity, Cryonics, Immortality). More dedication to spacially vast issues (Singularity, X-risk, Transhumanism).   

Less dedication to the parts of one's self that have a shorter life-span.  Such as the instantaneous gratification of philosophical traditions of the east (buddhism, hinduism) and some hedonistic traditions of the west (psychedelism, selfish instantaneous hedonism, sex and masturbation-ism, drugs-isms, thrill-isms). 

Also less dedication to time spans such as three months. Personal projects visible, completable and doable in such scales. 

This process of letting your past decisions trump your current decisions/feelings/emotions/intuitions was very fruitful for me, and for very long I thought (and still think) it made my life greater than the life of most around me (schoolmates, university peers, theater friends etc... not necessarily the people I choose to hang out with, after all, I selected those!). 

At some point more recently, and I'm afraid this might happen to the Effective Altruist community and the immortalist community of Less Wrong, I started feeling overwhelmed, a slave of "past me". Even though a lot of "past me" orders were along the lines of "maximize other people's utility, help everyone the most regardless of what those around you are doing".

Then the whole edifice crumbled, and I took 2 days off of all of life to go to a hotel in the woods and think/write alone to figure out what my current values are. 

I wrote several pages, thought about a lot of things. More importantly, I quantified the importance I give to different time-spans of my self (say 30 points to life-goals, 16 points to instantaneous gratification, 23 points to 3MonthGoals etc...). I also quantified differently sized circles of altruism/empathy  (X points for immediate family, Y points for extended family, Z points for near friends, T points for smart people around the globe, U points for the bottom billion, K points for aliens, A points for animals etc...). 

Knowing my past commitment to past selves, I'd expect these new quantificatonal regulatory forces I had just created to take over me, and cause me to spend my time in proportion to their now known quantities. In other words, I allowed myself a major change, a rewriting which dug deeper into my source code than previous re-writings. And I expected the consequences to be of the same kind than those previous re-writings. 

Seems I was wrong. I've become unstable. Trying to give an outside description the algorithm as it feels from the inside, it seems that the natural order of attention allocation which I had, like a blacksmith, annealed over the years, has crumbled. Instead, I find myself being prone to an evolutionary fight between several distinct desires of internal selves. A mix of George Ainslie's piconomics and plain neural darwinism/multiple drafts. 

Such instability, if not for anything else, for hormonal reasons, is bound not to last long. But thus far it carried me into Existentialism audiobooks, considering Vagabonding lifestyle as an alternative to a Utilitarian lifestyle, and considering allowing a personality dissolution into whatever is left of one's personality when we "allow it" (emotionally) to dissolve and reforge itself. 

The instability doesn't cause anxiety, sadness, fear or any negative emotion (though I'm at the extreme tail of the happiness setpoint, the equivalent in happiness of having an IQ 145, or three standard deviations). Contrarywise. It is refreshing and gives a sense of freedom and choice. 

This post can be taken to be several distinct things for different readers. 

1) A warning for utilitarian life-style people that allowing deep changes causes an instability which you don't want to let your future self do. 

2) A tale of a self free of past enslavery (if only for a short period of time), who is feeling well and relieved and open to new experiences. That is, a kind of unusual suggestion for unusual people who are in an unusual time of their lives. 

(Note: because of the unusual set-point thing, positive psychology advice should be discarded as a basis for arguments, I've already achieved ~0 marginal returns after 2000pgs of it)

3) This is the original intention of writing: I wanted to know the arguments in favor of a selfish vagabonding lifestyle, versus the arguments in favor of the Utilitarian lifestyle, because this is a particularly open-minded moment in my life, and I feel less biased than in most other times. For next semester, assume money is not an issue (both Vagabond and Utililtarian are cheap, as opposed to "you have a million dollars"). So, what are the arguments you'd use to decide that yourself? 

I'd like to divide three classes of reasons to read a discipline:

1) You are curious and want to begin reading by something 100-500 pages. I'd go for Pinker's 1990's  "How the mind works"

2) You want to screen the whole field, by reading something 500-1500 pages. I definitely recommend David Buss 2004 "The Handbook of Evolutionary Psychology" which defeats the usual SI recommendations on the field

3) You want to know the state of the art of the field, so you really need something that is very recent, say from the last 2 or 3 years at most.  This is me. Please help me if you know what should I read.  300-1500 seems a good interval.

Just for a comparative, in Cognitive Neuroscience, 3 would be 2009 "MIT The Cognitive Neurosciences IV"

 

Post your opinions on what 1 2 and 3 should be for Evolutionary Psychology.

Oh, and if you like Evolutionary Cognitive Neuroscience (a field so new I don't know any of the 3) please post yours too...

The CEV Sequence Summary: The CEV sequence consists of three posts tackling important aspects of CEV. It covers conceptual, practical and computational problems of CEV's current form. On What Selves Are draws on analytic philosophy methods in order to clarify the concept of Self, which is necessary in order to understand whose volition is going to be extrapolated by a machine that implements the CEV procedure. Troubles with CEV part1 and Troubles with CEV part2 on the other hand describe several issues that will be faced by the CEV project if it is actually going to be implemented. Those issues are not of conceptual nature. Many of the objections shown come from scattered discussions found on the web. Finally, six alternatives to CEV are considered.

 

Troubles with CEV Summary: Starting with a summary of CEV, we proceed to show several objections to CEV. First, specific objections to the use of Coherence, Extrapolation, and Volition. Here Part1 ends. Then, in Part2, we continue with objections related to the end product of performing a CEV, and finally, problems relating to the implementation of CEV. We then go on with a praise of CEV, pointing out particular strengths of the idea. We end by showing six alternatives to CEV that have been proposed, and considering their vices and virtues.

Meta: I think Troubles With CEV Part1 and Part2 should be posted to Main. So on the comment section of Part2, I put a place to vote for or against this upgrade.

 

Troubles with CEV Part2

 

5) Problems with the end product

5a) Singleton Objection. Even if all goes well and a machine executes the coherent extrapolated volition of humanity, the self modifying code it is running is likely to become the most powerful agent on earth (including individuals, governments, industries and other machines) If such a superintelligence unfolds, whichever goals it has (our CE volitions) it will be very capable of implementing. This is a singleton scenario. A singleton is “[T]he term refers to a world order in which there is a single decision-making agency at the highest level. Among its powers would be (1) the ability to prevent any threats (internal or external) to its own existence and supremacy, and (2) the ability to exert effective control over major features of its domain (including taxation and territorial allocation).”. Even though at first sight the emergence of a singleton looks totalitarian, there is good reason to establish a singleton as opposed to several competing superintelligences. If a singleton is obtained, the selective process of genetic and cultural evolution meets with a force that can counter its own powers. Something other than selection of the fittest takes place as the main developer of the course of history. This is desirable for several reasons. Evolution favors flamboyant displays, malthusian growth and in general a progressively lower income, with our era being an exception in its relative abundance of resources. Evolution operates on many levels (genes, memes, individuals, institutions, groups) and there is conflict and survival of the fittest in all of them. If evolution were to continue being the main driving force of our society there is great likelihood that several of the things we find valuable would be lost. Much of what we value has evolved as signaling (dancing, singing, getting jokes) and it is likely that some of that costly signaling would be lost without a controlling force such as a singleton. For this reason, having a singleton can be considered a good result in the grand scheme of things, and should not constitute worry to the CEV project, despite initial impressions otherwise. In fact if we do not have a singleton soon we will be Defeated by Evolution at the fastest level where evolution is occurring. At that level, the fast growing agents gradually obtain the resources of the remaining desirable agents until all resources are taken and desirable agents become extinct.

 

6) Problems of implementation

6a) Shortage Objections. To extract coherent extrapolated volitions from people seems to be not only immensely complicated but also computationally costly. Yudkowsky proposes in CEV that we should let this initial dynamic run for a few minutes and then redesign its machine, implementing the code it develops once it is mature. But what if maturity is not achieved? What if the computational intractability of muddled concepts and spread overwhelm the computing capacity of the machine, or exceed the time it is given to process it's input?

6b) Sample bias. The CEV machine implements the volition of mankind, such is the suggestion. But from what sample of people will it extrapolate? Certainly it will not do a fine grained reading of everyone's brainstates in order to start operating, it will more likely extrapolate from sociological, anthropological and psychological information. Thus its selection of groups extrapolated will matter a lot in the long run. It may try to correct sampling bias by obtaining information about other cultures (besides programmers culture and whichever other cultures it starts with), but the vastness of human societal variation can be a hard challenge to overcome. We want to fairly take into account everyone's values, rather than privileging those of the designers.

6c) The Indeterminacy Objection. Suppose we implement the CEV of a group of people including three catholics, a muslim and two atheists, all of them English speakers. What if the CEV machine fails to consider the ethical divergence of their moral judgments by changing the meaning of the word 'god'? While extrapolating, many linguistic tokens (words) will appear (e.g. as parts of ethical imperatives). Since Quine's (1960) thesis of indeterminacy of reference, we know that the meanings of words are widely under-determined by their usage. A machine that reads my brainstate looking for cues on how to CEV may find sufficiently few mentions of a linguistic token such as 'god' that it ends up able to attribute almost any meaning to it (analogous to Löwenheim-Skolem theorem), and it may end up tampering with the token's meaning for the wrong reasons (to increase coherence at cost of precision).

 

7) Praise of CEV

7a) Bringing the issue to practical level

Despite all previous objections, CEV is a very large reduction in the problem space of how to engineer a nice future. Yudkowsky's approach is the first practical suggestion for how an artificial moral agent might do something good, as opposed to destroying humanity. Simply starting the debate of how to implement an ethical agent that is a machine built by humans is already a formidable achievement. CEV sets the initial grounding above which will be built stronger ideas for our bright future.

7b) Ethical strength of egalitarianism

CEV is a morally egalitarian ethically designed theory. Each current human stands in the same quantitative position relative to how much his volition will contribute to the final sum. Even though the CEV implementing machine will only extrapolate some subset of humans, it will try to make that subset in as much as possible a political representative of the whole.

 

8) Alternatives to CEV

8a) The Nobel Prize CEV

Here the suggestion is to do CEV on only a subset of humanity (which might be necessary anyway for computational tractability). Phlebas asks:

“[Suppose] you had to choose a certain subset of minds to participate in the initial dynamic?

What springs to my mind is Nobel Prize winners, and I suspect that this too is a Schelling point. This seems like a politically neutral selection of distinguished human beings (particularly if we exclude the Peace Prize) of superlative character and intellect.”

In the original CEV, the initial dynamic would have to either scan all brains (unlikely) or else extrapolate predictions made with its biological, sociological, anthropological and psychological resources from a subset of brains, correcting for all correctable biases in its original sample. This may be a very daunting task; It may just be easier to preselect a group and extrapolate their volition. Which computational procedures would you execute in order to be able to extrapolate a set of Jews and Arabs if your initial sample were only composed of Jews? That is, how can you predict extrapolated Arabs from Jews? This would be the level of difficulty of the task we impose on CEV if we let the original dynamic scan only western minds and try to extrapolate Pirahã, Maori, Arab, and Japanese minds out of this initial set. Instead of facing this huge multicultural demand, using Nobel winners wouldn't detract away from the initial mindset originating the CEV idea. The trade-off here is basically between democracy in one hand and tractability on the other. Still Phlebas: “I argue that the practical difficulty of incorporating all humans into the CEV in the first place is unduly great, and that the programming challenge is also made more difficult by virtue of this choice. I consider any increase in the level of difficulty in the bringing into existence of FAI to be positively dangerous, on account of the fact that this increases the window of time available for unscrupulous programmers to create uFAI. “

8b) Building Blocks for Artificial Moral Agents

In his article “Building Blocks for Artificial Moral Agents” Vincent Wiegel provides several interesting particularities that must be attended to when creating these agents: “An agent can have as one of its goals or desires to be a moral agent, but never as its only or primary goal. So the implementation of moral reasoning capability must always be in the context of some application in which it acts as a constraint on the other goals and action.” Another: “[O]nce goals have been set, these goals must have a certain stickiness. Permanent goal revision would have a paralyzing effect on an agent and possibly prevent decision making.” Even though his paper doesn't exactly provide a substitute for CEV, it provides several insights into the details that must be taken in consideration when implementing AGI. To let go of the user-friendly interface that the CEV paper has and to start thinking about how to go about implementing moral agents on a more technical ground level I suggest examining his paper as a good start.

8c) Normative approach

A normative or deontological approach would have the artificial agent following rules, that is, telling it what is or not allowed. Examples of deontological approaches are Kant's maxim, Gert's ten principles in Morality and Asimov's three laws of robotics. A normative approach doesn't work because there are several underdeterminations in telling the agent what not to do, trillions of subtle ways to destroy everything that matters without breaking any specific set of laws.

8d) Bottom up approaches

8d.1) Associative Learning

There are two alternatives to CEV that would build from the bottom up, the first is associative learning implemented by a neural network reacting to moral feedback, and the second evolutionary modeling of iterated interacting agents until the cusp of emergence of “natural” morality. In the first approach, we have a neural network learning morality like children were thought to learn in the good old blank slate days, by receiving moral feedback under several different contexts and being rewarded or punished according to societal rules. The main advantage here is tractability, algorithms for learning associatively are known and tractable thus rendering the entire process computationally viable. The disadvantage of this approach is inscrutability, we have no clear access to where within the system the moral organ is being implemented. If we cannot scrutinize it we wouldn't be able to understand eventual failures. Just one possible failure will suffice to show why bottom up associative approaches are flawed, that is the case in which an AGI learns a utility function ascribing utility to individuals self-described as 10 in their happiometers. This of course would tile the universe with sets of particles vibrating as little as possible to say “I'm happy ten” over and over again.

8d.2) Artificial Evolution

The second bottom up approach consists of evolving morality from artificial life forms. As is known, morality (or altruism) will evolve once iterated game theoretic scenarios of certain complexity start taking place in an evolving system of individuals. Pure rationality guides individuals into being nice merely because someone might be nice in return, or as Dawkins puts it, nice guys finish first. The proposal here would then be that we let artificial life forms evolve to the point where they become moral, and once they do, input AGI powers into those entities. To understand why this wouldn't work, let me quote Allen, Varner and Zinzer “ In scaling these environments to more realistic environments, evolutionary approaches are likely to be faced with some of the same shortcomings of the associative learning approaches : namely that sophisticated moral agents must also be capable of constructing an abstract, theoretical conception of morality.” If we are to end up with abstract theories of morality, a safer path would be to inscribe the theories to begin with, minimizing the risk of ending up with lower than desirable level of moral discernment. I conclude that bottom up approaches, by themselves, provide insufficient insight as to how to go about building an Artificial Moral Agent such as the one CEV proposes.

8e) Hybrid holonic ("Holonic" is a useful word to describe the simultaneous application of reductionism and holism, in which a single quality is simultaneously a combination of parts and a part of a greater whole [Koestler67]. Note that "holonic" does not imply strict hierarchy, only a general flow from high-level to low-level and vice versa.  For example, a single feature detector may make use of the output of lower-level feature detectors, and act in turn as an input to higher-level feature detectors.  The information contained in a mid-level feature is then the holistic sum of many lower-level features, and also an element in the sums produced by higher-level features.)

 A better alternative than any of the bottom up suggestions is to have a hybrid model with both deontological and bottom up elements. Our morality is partly hardwired and mostly software learning so that we are hybrid moral systems. A hybrid system may for instance be a combination of thorough learning of moral behavior by training plus Gert's set of ten moral principles. The advantage of hybrid models is that they combine partial scrutability with bottom up tractability and efficiency. In this examination of alternatives to CEV a Hybrid Holonic model is the best contestant and thus the one to which our research efforts should be directed.

 

8f) Extrapolation of written desires

Another alternative to CEV would be to extrapolate not from reading a brain-state, but from a set of written desires given by the programmers. The reason for implementing this alternative would be the technical non-feasibility of extrapolating from brain states. That is, if our Artificial General Intelligence is unable to read minds but can comprehend language. We should be prepared for this very real possibility since language is countless times simpler than active brains. To extrapolate from the entire mind is a nice ideal, but not necessarily an achievable one. To consider which kinds of desires should be written in such case is beyond the scope of this text.

8g) Using Compassion and Respect to Motivate an Artificial Intelligence.

Tim Freeman proposes what is to my knowledge the most thorough and interesting alternative to CEV to date. Tim builds up from Solomonoff induction, Schmidhuber's Speed Prior and Hutters AIXI to develop an algorithm that infers people's desires from their behavior. The algorithm is exposed in graphic form, in Python and in abstract descriptions in English. Tim's proposal is an alternative to CEV because it does not extrapolate people's current volition, thus it could only be used to produce a CV, not a CEV. His proposal deserves attention because it does, unlike most others, take in consideration the Friendly AI problem, and it actually comes with an implementation (though idealized) of the ideas presented in the text, unlike CEV. By suggesting a compassion coefficient and a (slightly larger) respect coefficient, Tim is able to solve many use cases that any desirable and friendly AGI will have to solve, in accordance to what seems moral and reasonable from a humane point of view. The text is insightful, for example, to solve wire-heading, it suggests: “The problem here is that we've assumed that the AI wants to optimize for my utility applied to my model of the real world, and in this scenario my model of the world diverges permanently from the world itself. The solution is to use the AI's model of the world instead. That is, the AI infers how my utility is a function of the world (as I believe it to be), and it applies that function to the world as the AI believes it to be to compute the AI's utility.“ It appears to me that just as any serious approach to AGI has to take in consideration Bayes, Speed Prior and AIXI, any approach to the problem that CEV tries to solve will have to consider Tim's “Using Compassion and Respect to Motivate an Artificial Intelligence” at some point, even if only to point out its mistakes and how they can be solved by posterior, more thoroughly devised algorithms. In summary, even though Tim's proposal is severely incomplete, in that it does not describe all, or even most steps that an AI must take in order to infer intentions from behavior, it is still the most complete work that tries to tackle this particular problem, while at the same time worrying about Friendliness and humaneness.

 

Studies related to CEV are few, making each more valuable, some topics that I have not had time to cover, but would like to suggest to prospective researchers are:

Solvability of remaining problems

Historical perspectives on problems

Likelihood of solving problems before 2050

How humans have dealt with unsolvable problems in the past

The CEV Sequence Summary: The CEV sequence consists of three posts tackling important aspects of CEV. It covers conceptual, practical and computational problems of CEV's current form. On What Selves Are draws on analytic philosophy methods in order to clarify the concept of Self, which is necessary in order to understand whose volition is going to be extrapolated by a machine that implements the CEV procedure. Troubles with CEV part1 and Troubles with CEV part2 on the other hand describe several issues that will be faced by the CEV project if it is actually going to be implemented. Those issues are not of conceptual nature. Many of the objections shown come from scattered discussions found on the web. Finally, some alternatives to CEV are considered.

 

Troubles with CEV Summary: Starting with a summary of CEV, we proceed to show several objections to CEV. First, specific objections to the use of Coherence, Extrapolation, and Volition. Here Part1 ends. Then, in Part2, we continue with objections related to the end product of performing a CEV, and finally, problems relating to the implementation of CEV. We then go on with a praise of CEV, pointing out particular strengths of the idea. We end by showing six alternatives to CEV that have been proposed, and considering their vices and virtues.

Meta: I think Troubles With CEV Part1 and Part2 should be posted to Main. So on the comment section of Part2, I put a place to vote for or against this upgrade.

 

Troubles with CEV Part1

 

Summary of CEV

To begin with, let us remember the most important slices of Coherent Extrapolated Volition (CEV).

“Friendly AI requires:

1.  Solving the technical problems required to maintain a well-specified abstract invariant in a self-modifying goal system. (Interestingly, this problem is relatively straightforward from a theoretical standpoint.)

2.  Choosing something nice to do with the AI. This is about midway in theoretical hairiness between problems 1 and 3.

3.  Designing a framework for an abstract invariant that doesn't automatically wipe out the human species. This is the hard part.

But right now the question is whether the human species can field a non-pathetic force in defense of six billion lives and futures.”
“Friendliness is the easiest part of the problem to explain - the part that says what we want. Like explaining why you want to fly to London, versus explaining a Boeing 747; explaining toast, versus explaining a toaster oven. ”

“To construe your volition, I need to define a dynamic for extrapolating your volition, given knowledge about you. In the case of an FAI, this knowledge might include a complete readout of your brain-state, or an approximate model of your mind-state. The FAI takes the knowledge of Fred's brainstate, and other knowledge possessed by the FAI (such as which box contains the diamond), does... something complicated... and out pops a construal of Fred's volition. I shall refer to the "something complicated" as the dynamic.”

This is essentially what CEV is: extrapolating Fred's mind and everyone else's in order to grok what Fred wants. This is performed from a reading of Fred's psychological states, be it through unlikely neurological paths, or through more coarse grained psychological paths. There is reason to think that a complete readout of a brain is overwhelmingly more complicated than a very good descriptive psychological approximation. We must make sure though that this approximation does not rely on our common human psychology to be understood. The descriptive approximation has to be understandable by AGI's, not only by evolutionarily engineered humans. Continuing the summary.

“In poetic terms, our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.“

“Had grown up farther together: A model of humankind's coherent extrapolated volition should not extrapolate the person you'd become if you made your decisions alone in a padded cell. Part of our predictable existence is that we predictably interact with other people. A dynamic for CEV must take a shot at extrapolating human interactions, not just so that the extrapolation is closer to reality, but so that the extrapolation can encapsulate memetic and social forces contributing to niceness.“

“the rule [is] that the Friendly AI should be consistent under reflection (which might involve the Friendly AI replacing itself with something else entirely).”

“The narrower the slice of the future that our CEV wants to actively steer humanity into, the more consensus required.“

“The dynamic of extrapolated volition refracts through that cognitive complexity of human minds which lead us to care about all the other things we might want; love, laughter, life, fairness, fun, sociality, self-reliance, morality, naughtiness, and anything else we might treasure. ”

“It may be hard to get CEV right - come up with an AI dynamic such that our volition, as defined, is what we intuitively want. The technical challenge may be too hard; the problems I'm still working out may be impossible or ill-defined.”

“The same people who aren't frightened by the prospect of making moral decisions for the whole human species lack the interdisciplinary background to know how much complexity there is in human psychology, and why our shared emotional psychology is an invisible background assumption in human interactions, and why their Ten Commandments only make sense if you're already a human. ”

“Even if our coherent extrapolated volition wants something other than a CEV, the programmers choose the starting point of this renormalization process; they must construct a satisfactory definition of volition to extrapolate an improved or optimal definition of volition. ”

 

Troubles with CEV

1) Stumbling on People, Detecting the Things CEV Will Extrapolate:

Concepts on which CEV relies that may be ill-defined, not having a stable consistent structure in thingspace.

CEV relies on many concepts, most notably the concepts of coherence, extrapolation and volition. We will discuss the problems of coherence and extrapolation shortly, for now I'd like to invoke a deeper layer of conceptual problems regarding the execution of a CEV implementing machine. A CEV executing machine ought to be able to identify the kind of entities whose volitions matter to us, the machine must be able to grasp selfhood, or personhood. The concepts of self and person are mingled and complex, and due to their complexity I have dedicated a separate text to address the issue of incompleteness, anomalousness, and fine-grainedness of selves.

 

2) Troubles with coherence

2a) The Intrapersonal objection: The volitions of the same person when in two different emotional states might be different - it’s as if they are two different people. Is there any good criteria by which a person’s “ultimate” volition may be determined? If not, is it certain that even the volitions of one person’s multiple selves will be convergent? As explained in detail in Ainslie's “Breakdown of Will”, we are made of lots of tinier interacting time-slices whose conflicts cannot be ignored. My chocolate has value 3 now, 5 when it's in my mouth and 0 when I reconsider how quick the pleasure was and how long the fat will stay. Valuations not only interpersonally, but also intrapersonally conflict. The variation in what we value can be correlated with not only with different distances in time, but also different emotional states, priming, background assumptions and other ways in which reality hijacks brains for a period.

 

2b) The Biological Onion objection: Our volitions can be thought of to be like an onion, layers upon layers of beliefs and expectations. The suggestion made by CEV is that when you strip away the layers that do not cohere, you reach deeper regions of the onion. Now, and here is the catch, what if there is no way to get coherence unless you stripe away everything that is truly humane, and end up being left only with that which is biological. What if in service of coherence we end up stripping away everything that matters and end up only with our biological drives? There is little in common between Eliezer, Me and Al Qaeda terrorists, and most of it is in the so called reptilian brain. We may end up with a set of goals and desires that are nothing more than “Eat Survive Reproduce,” which would qualify as a major loss in the scheme of things. In this specific case, what ends up dominating CEV is what evolution wants, not what we want. Instead of creating a dynamic with a chance of creating the landscape of a Nice Place to Live, we end up with some exotic extrapolation of simple evolutionary drives. Let us call this failure mode Defeated by Evolution. We are Defeated by Evolution if at any time the destiny of earth becomes nothing more than darwinian evolution all over again, at a different level of complexity or at different speed. So if CEV ends up stripping the biological onion of its goals that matter, extrapolating only a biological core, we are defeated by evolution.

 

3) Troubles with extrapolation

3a) The Small Accretions Objection: Are small accretions of intelligence analogous to small accretions of time in terms of identity? Is extrapolated person X still a reasonable political representative of person X? Are X's values desirably preserved when she is given small accretions of intelligence? Would X allow her extrapolation to vote for her?

This objection is made through an analogy. For countless time philosophers have argued about the immortality of the soul, the existence of the soul, the complexity of the soul and last but not least the identity of the soul with itself over time.

Advancements in the field of philosophy are sparse and usually controversial, and if we were depending on a major advance in understanding of the complexity of our soul we'd be in a bad situation. Luckily, our analogy relies on the issue of personal identity, where it appears as though the issue of personal identity has been treated in sufficient detail by the book Reasons and Persons, Derek Parfit's major contribution to philosophy: Covering cases from fission and fusion to teleportation and identity over time. It is identity over time which concerns us here; Are you the same person as the person you were yesterday? How about one year ago? Or ten years? Derek has helped the philosophical community by reframing the essential question, instead of asking whether X is the same over time, he asks if personal identity is what matters, that is, that which we want to preserve when we deny others the right of shooting us. More recently he develops the question in full detail in his “Is Personal Identity What Matters?”(2007) a long article were all the objections to his original view are countered in minute detail.

We are left with a conception of identity over time not being what matters, and psychological relatedness being the best candidate to take its place. Personal identity is dissolved into a quantitative, not qualitative, question. How much are you the same and the one you were yesterday? Here some percentage enters the field, and once you know how much you are like the person you were yesterday, there is no further question about how much you are the person you were yesterday. We had been asking the wrong question for long, and we risk to be doing the same thing with CEV. What if extrapolation is a process that dissolves that which matters about us and our volitions? What if there is no transitivity of what matters between me and me+1 or me+2 in the intelligence scale? Then abstracting my extrapolation will not preserve what had to be preserved in the first place. To extrapolate our volition in case we knew more, thought faster and had grown up farther together is to accrue small quantities of intelligence during the dynamic, and doing this may be risky. Even if some of our possible extrapolations would end up generating part of a Nice Place to Be, we must be sure none of the other possible extrapolations actually happen. That is, we must make sure CEV doesn't extrapolate in a way that for each step of extrapolation, one slice of what matterness is lost. Just like small accretions of time make you every day less the person you were back in 2010, maybe small accretions of intelligence will be displacing ourselves from what is preserved. Maybe smarter versions of ourselves are not us at all - this is the The Small Accretions Objection.

4) Problems with the concept of Volition

4a) Blue minimizing robots (Yvain post)

4b) Goals vs. Volitions

The machine's actions should be grounded in our preferences, but those preferences are complex and opaque, making our reports unreliable; to truly determine the volitions of people, there must be a previously recognized candidate predictor. We test the predictor in its ability to describe current humans volitions before we give it the task of comprehending extrapolated human volition.

4c) Want to want vs. Would want if thought faster, grew stronger together

Eliezer suggests in CEV that we consider a mistake to give Fred box A if he wanted box A while thinking it contained a diamond, in case we know both that box B contains the diamond and that Fred wants the diamond. Fred's volition, we are told, is to have the diamond, and we must be careful to create machines that extrapolate volition, not mere wanting. This is good, but not enough. There is a sub-area of moral philosophy dedicated to understanding that which we value, and even though it may seem at firsthand that we value our volitions, the process that leads from wanting to having a volition is a different process than the one that leads from wanting to having a value. Values, as David Lewis has argued, are what we want to want. Volitions on the other hand are what we would ultimately want under less stringent conditions. Currently CEV does not consider the iterated wantness aspect of things we value (the want to want aspect). This is problematic in case our volitions do not happen to be constrained by what we value, that is, what we desire to desire. Suppose Fred knows that the diamond he thinks is in box A comes from a bloody conflict region. Fred hates bloodshed and he truly desires not to have desires for diamonds, he wants to be a person that doesn't want diamonds from conflict regions. Yet the flesh is weak and Fred, under the circumstance, really wants the diamond. Both Fred's current volition, and Fred's extrapolated volition would have him choose box B, if only he knew, and in neither case Fred's values have been duly considered. It may be argued that a good enough extrapolation would end up considering his disgust of war, but here we are talking not about a quantitative issue (how much improvement there was) but a qualitative leap (what kind of thing should be preserved). If it is the case, as I argue here, that we ought to preserve what we want to want, this must be done as a separate consideration, not as an addendum, to preserving our volitions, both current and extrapolated.

 

Continues in Part2

Hi anglophones, this topic is only for brazilians, so someone may post in portuguese and part of this is in portuguese (We will translate it to english if necessary when the time comes).

Hello Brazilians, I'm creating this topic because some misallocated questions were posed on this one.

First, the numbers.In Less Wrong:

Me, Gust, Paulovsk, zecaurubu, Gracunha, Mexamark, dyokomizo.

From IERFH (Instituto Ética, Racionalidade, e o Futuro da Humanidade)

Leo Arruda, João Fabiano, me, Pierre , Jonatas Muller, Pablo,  Lauro (paralelo), Rafael, plus 3 others.

This makes us at least 17, almost one every 10 million people (not great...)

Some of us are in São Paulo. About 10.

Eventually, this topic may attract some others, and we can create a meeting.

Now, regarding the question a few of you did, IERFH's mission:

Gerar alto impacto positivo no longo prazo, produzindo conhecimentos e reunindo pessoas que contribuam para melhor pensar as questões éticas que irão definir o futuro da humanidade.

About: Somos um time comprometido com fazer o mundo melhor, agora e no futuro. Para isso, estamos reunindo a comunidade brasileira de racionalistas, utilitaristas, transhumanistas, e outros entusiastas e tranformando ideais e teorias em ações. Ao lado de grandes organizações internacionais de caridade, tecnologia e ética, nos propomos a ser o vetor dos esforcos brasileiros nesses campos. O IERFH opera em 3 frentes: o estudo do que é bom e deve ser buscado e preservado: a Ética Pura e Aplicada; as maneiras mais eficientes de raciocínio, tomada de decisões e os seus erros mais comuns: a Racionalidade Epistêmica e Prática; e por fim como aplicar estes campos para garantir a plena realização de todo o potencial humano: o Futuro da Humanidade 

 

Racionalidade: Ser racional é conseguir conquistar, com pouco dispêndio de recursos, aquele que se deseja, entre todos os cenários possíveis que poderiam ter ocorrido. Racionalidade epistêmica é a capacidade de entender com o mundo atual; Racionalidade prática, a capacidade de guiar o mundo atual em direção ao mundo desejado. Para ser racional, duas capacidades são fundamentais, a capacidade de desviar dos bias cognitivos, falhas sistemáticas da nossa cognição, e permitir que o conhecimento adquirido atinja todos os campos de nosso conhecimento, integrando a informação aprendida e garantindo que ela tenha um efeito proporcional em nossas vidas. Essa comunidade do IERFH pretende nos guiar nesse sentido.

Futuro da Humanidade: Para guiar o futuro da humanidade numa direção desejável, é necessário, antes de tudo, desviar dos grandes riscos catastróficos de origem tecnológica que estamos criando conforme criamos novas tecnologias. Para tal, é também necessário compreender e corrigir os bias cognitivos aos quais nossos cérebros estão propensos. Finalmente, garantidas a segurança de nossos valores fundamentais, e corrigidos os nossos desvios de racionalidade, podemos seguir adiante na realização de todo o potencial futuro humano, através de biotecnologia, nanotecnologia, inteligência artificial e coordenação global.

The missing part "Ética" isn't written yet. But I think you get the general idea.Think Bostrom, think Utilitarianism.

Se acharam interessante a descrição, talvez gostem desse post. Ou, para os familiarizados com Yudkowsky desse, sobre CEV.

 

We are developing our website, that is why we still don't have one.

Given the broadness of our scope, we are, of course, in need of new people (specially to translate) but we will only post to less wrong about the group in detail (in english) in a few months. If you are interested, contact me in the private message section.  We meet on skype, and rarely in Pinheiros, São Paulo.

I hope we can start to form a Brazilian rationalist community, both of less wrongers general, and within IERFH and thank Gust for the initiative of creating the meeting topic that made me write this one.

Hi, I've been trying to publish an article to less wrong for about a week, but I'm unable to copy the text to the post area and submit it.

It says: Submitting, then it stops and nothing happened.

When I sliced the entire text in small parts (all copied from Open Office) I manage to publish drafts, some with errors such as missing spaces.

When I re-copy from the drafts to another, bigger draft, then many spaces become missing, and part of the text is aligned to the right border, part isnt.

The text was mostly written in open office. I would like help from someone who has a normal Office. If you post me a message with your e-mail in the private message area, I can send it to you, so you see if you can either publish it in drafts, and then copy it again to me in a publishable form, or edit somethign in office that I'm unable in open office, and send the file back to me so I can publish.

I know this is asking a lot, and I would be thankful for anyone who helps me out of this conundrum.