Finally did it. I'd like exactly 7 karma please.

For the goal of eventually creating FAI, it seems work can be roughly divided into making the first AGI (1) have humane values and (2) keep those values. Current attention seems to be focused on the 2nd category of problems. The work I've seen in the first category: CEV (9 years old!), Paul Christiano's man-in-a-box indirect normativity, Luke's decision neuroscience, Daniel Dewey's value learning... I really like these approaches but they are only very early starting points compared to what will eventually be required.

Do you have any plans to tackle the humane values problem? Do MIRI-folk have strong opinions on which direction is most promising? My worry is that if this problem really is as intractable as it seems, then working on problem (2) is not helpful, and our only option might be to prevent AGI from being developed through global regulation and other very difficult means.

I think I remember hearing Holden Karnofsky give a different version of the quote in the second footnote. Something like: It's like there's not just one set of dropped keys, but there are keys everywhere. Some are in the dark, some are under the light. And you don't know where your keys are, but you have no chance of finding them in the dark, so you might as well look at the ones in the light where you have a chance of finding them. (Meaning: look at charities/interventions you have data on, because while there are probably better ones out there without data supporting them, you'll never be able to identify which ones they are.)

Thanks, this is what I tried to say. Reducing suffering is far, eating well is near.

Also, if a book or a website comes with vegetarian/vegan propaganda, I would assume those people are likely to lie or exaggerate. No propaganda -- no suspicion.

This may be just about vegetarians around me, but often people who are into vegetarianism are also into other forms of food limitations, so I often find their food unappealing. They act like an anti-advertisement to vegetarian food. (Perhaps there is an unconscious status motive here: the less people join them, the more noble they are. Which is not how an effective altruist should think.) On the other hand, when I go to some Indian or similar ethnic restaurant, I love the food. It tastes well, it has different components and good spice. I mean, what's wrong about using spice? If your goal is to reduce animal suffering, nothing. But if your goal is to have a weirdest diet possible (no meat, no cooking, no taste, everything compatible with the latest popular book or your horoscope), spice is usually on the list of forbidden components.

In short, vegetarianism is often not about not eating animals. So if you focus on "good meal (without meat)" part, and ignore the vegetarianism, you may win people like me. Even if I don't promise to give up meat completely, I can reduce its consumption simply because tasty meals without meat outcompete tasty meals with meat on my table.

The main reason to focus on existential risk generally, and human extinction in particular, is that anything else about posthuman society can be modified by the posthumans (who will be far smarter and more knowledgeable than us) if desired, while extinction can obviously never be undone. For example, any modification to the English language, the American political system, the New York Subway or the Islamic religion will almost certainly be moot in five thousand years, just as changes to Old Kingdom Egypt are moot to us now.

The only exception would be if the changes to post-human society are self-reinforcing, like a tyrannical constitution which is enforced by unbeatable strong nanotech for eternity. However, by Bostrom's definition, such a self-reinforcing black hole would be an existential risk.

Are there any examples of changes to post-human society which a) cannot ever be altered by that society, even when alteration is a good idea, b) represent a significant utility loss, even compared to total extinction, c) are not themselves total or near-total extinction (and are thus not existential risks), and d) we have an ability to predictably effect at least on par with our ability to predictably prevent x-risk? I can't think of any, and this post doesn't provide any examples.

Hiding animal suffering probably makes us "more ethical". Vegetarians are usually people who were raised in cities. People who grew up slaughtering animals often can't even comprehend that someone else can have a problem with it.

Here is my solution to the personal identity issues, and I don't think it really violates common intuitions too badly. ...................

Woah, look, I* exist! Check out all this qualia! I'm having thoughts and sensations. Hm.... among my qualia is a set of memories. Instincts, intuition, and knowledge about how things work. Oh, neat, among those intuitions is a theoretical model of the universe! I hope it is accurate...well anyway it's the most appealing model I've got right now.

In an instant, I will disappear forever. I have a vague notion that this idea aught to be terrifying, but my utility function just sorta shrugs as terror completely fails to flow through my veins. I don't care that I'm going to disappear...but here is what I do care about - my model of the universe has informed me that everything that I'm doing right now will leave a memory trace. In the next few moments, I will cease to exist and a being will appear who will remember most of what I am feeling right now. That being will then disappear and be replaced by another. This will continue for a long time.

I care about experiencing happiness right now, in this moment before I disappear forever. I also care about those future beings - I want them to experience happiness during the moment of their existence. too. It's sort of like altruism for future beings which will carry my trace, even though we all realize altruism isn't the right word. Maybe we can call it "self-altruism" or more colloquially, self love.

Before you cleverly suggest making an infinite number of copies of myself and pleasuring them, that's not the only thing my utility function cares about. I'm not entirely self-altruistic - I've currently got a pretty strong "don't create multiple redundant copies of sentient beings"utility component, or shall we say gut instinct.

........

*The use of the word "I" is convenient here, but I'm sure we all realize that we can deconstruct "personal identity" spatially as well as temporally.

Anyway, that's part of my current philosophical worldview, and I don't feel confused by any of the problems in the trilemma. Perhaps I'm not thinking about it carefully enough - can anyone point out a reason why I should be confused?

You might note that while I have not tabood subjective experience entirely, I have noted that an "individual" can only subjectively experience the present moment, and that "your" utility function compels "you" to act in such a way as to bring about your preferred future scenarios, in accordance with your (objective) model of the universe.

I guess I've essentially bitten the "reject all notions of a thread connecting past and future subjective experiences" bullet that Eliezer Y said he had trouble biting...but I think my example illustrates that "biting that bullet" does not result in an incoherent utility function, as EY stated in his post. I don't really think it's fair to call it a "bullet" at all.

Just think of the feeling of "subjective expectation" as the emotional, human equivalent to a utility function which factors in the desires of future beings that carry your memories. It's analogous to how love is the emotional equivalent to a utility function which takes other people's feelings into account

I frankly hope it represents the future of LessWrong.

What I do claim is that many specific philosophical positions and methods are undermined by scientific knowledge about how brains and other systems work.

It would be useful to have a list of such positons which are still taken seriously by anglophone philosophy. FYI Hegel, Heidegger, Berkely and other easy targets generally aren't.

For example, I've argued that a particular kind of philosophical analysis, which assumes concepts are defined by necessary and sufficient conditions, is undermined by psychological results showing that brains don't store concepts that way.

But science gains benefits from using categories that are artficially tidy compared to folk concepts.Thus a tomato is not a scientific vegetable, nor a whale a scientific fish. Why shouldn't philosophy do the same?

This seems to me like a major spot where the dualistic model of self-and-world gets introduced into reinforcement learning AI design (which leads to the Anvil Problem). It seems possible to model memory as part of the environment by simply adding I/O actions to the list of actions available to the agent. However, if you want to act upon something read, you either need to model this by having atomic read-and-if-X-do-Y actions, or you still need some minimal memory to store the previous item(s) read in.