Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Utilitarian 06 June 2013 05:37:20AM *  4 points [-]

Jonah, I agree with what you say at least in principle, even if you would claim I don't follow it in practice. A big advantage of being Bayesian is that you retain probability mass on all the options rather than picking just one. (I recall many times being dismayed with hacky approximations like MAP that let you get rid of the less likely options. Similarly when people conflate the Solomonoff probability of a bitstring with the shortest program that outputs it, even though I guess in that case, the shortest program necessarily has at least as much probability as all the others can combined.)

My main comment on your post is that it's hard to keep track of all of these things computationally. Probably you should try, but it can get messy. It's also possible that in keeping track of too many details, you introduce more errors than if you had kept the analysis simple. On many questions in physics, ecology, etc., there's a single factor that dominates all the rest. Maybe this is less true in human domains because rational agents tend to produce efficiencies due to eating up the free lunches.

So, I'm in favor of this approach if you can do it and make it work, but don't let the best be the enemy of the good. Focus on the strong arguments first, and only if you have the bandwidth go on to think about the weak ones too.

Comment author: Desrtopa 12 April 2013 12:45:26PM 8 points [-]

As far as we can tell these new vegetarians were eating like normal Americans before they saw the videos.

I suspect they were not. People who have a more entrenched habit requiring greater life alteration to change it are less likely to give it up.

Of the people I've known who went vegetarian, I don't think any of them went from being big meat eaters to totally abstaining in a single step.

Comment author: Utilitarian 13 April 2013 05:49:55AM 3 points [-]

I used to eat a lot of chicken and eggs before I read Peter Singer. After that, I went cold turkey (pardon the expression).

Comment author: ChristianKl 09 April 2013 03:12:16PM 2 points [-]

However, if the AGI is hiding itself, it could go a long way before people realized what was going on.

Exactly. On the one hand the AGI doesn't try to let humans get wind of it's plans. On the other hand it's going to produce distractions.

You have to remember how delusional some folks are. Imaging trying to convince the North Korean's that they have to destroy their computers because those computer are infested with an evil AI.

Even in the US nearly half of the population still believes in creationism. How many of them can be convinced that the evil government is trying to take away their computers to establish a dictatorship?

Before the government goes attempts to trash the computer the AI sent an email to a conspiracy theory website, where it starts revealing some classified documents it aquired through hacking that show government misbehavior.

Then it sents an email to the same group saying that the US government is going to shut down all civilian computers because freedom of speech is to dangerous to the US government and that the US government will be using the excuse that the computers are part of a Chinese botnet.

In our time you need computers to stock supermarket shelves with goods. Container ships need GPS and see charts to navigate.

People start fighting each other. Some are likely to blame the people who wanted to thrash the computers as responsible for the mess.

Even if you can imagine shutting of all computer in 2013, in 2033 most cars will be computers in which the AI can rest. A lot of military firepower will be in drones that the AI can control.

Comment author: Utilitarian 10 April 2013 11:22:22AM 0 points [-]

Some really creative ideas, ChristianKl. :)

Even with what you describe, humans wouldn't become extinct, barring other outcomes like really bad nuclear war or whatever.

However, since the AI wouldn't be destroyed, it could bide its time. Maybe it could ally with some people and give them tech/power in exchange for carrying out its bidding. They could help build the robots, etc. that would be needed to actually wipe out humanity.

Obviously there's a lot of conjunction here. I'm not claiming this scenario specifically is likely. But it helps to stimulate the imagination to work out an existence proof for the extinction risk from AGI.

Comment author: ikrase 08 April 2013 08:50:19PM 1 point [-]

I think he meant more along the lines of computers/robots/ non-super AIs becoming more powerful, IDK.

Comment author: Utilitarian 09 April 2013 02:02:34AM 1 point [-]

It's not at all clear that a AGI will be human-like, anyone than humans are dog-like.

Ok, bad wording on my part. I meant "more generally intelligent."

How do you fight the AGI past that point?

I was imagining people would destroy their computers, except the ones not connected to the Internet. However, if the AGI is hiding itself, it could go a long way before people realized what was going on.

Interesting scenarios. Thanks!

Comment author: ChristianKl 08 April 2013 12:13:45PM 2 points [-]

We don't necessarily see huge investments in AI safety yet, but this will probably change in time, as we begin to see more AIs that get out of control and cause problems on a local scale.

Once we see an out of control AI it's to late to do AI safety. Given current computer security the AI could hack itself into every computer in the world and resist easy shutdown.

When it comes to low probability high impact events waiting for small problem to cause awareness of the issue is just dangerous.

Comment author: Utilitarian 08 April 2013 01:10:45PM *  1 point [-]

As we begin seeing robots/computers that are more human-like, people will take the possibility of AGIs getting out of control more seriously. These things will be major news stories worldwide, people will hold natural-security summits about them, etc. I would assume the US military is already looking into this topic at least a little bit behind closed doors.

There will probably be lots of not-quite-superhuman AIs / AGIs that cause havoc along the road to the first superhuman ones. Yes, it's possible that FOOM will take us from roughly a level like where we are now to superhuman AGI in a matter of days, but this scenario seems relatively unlikely to me, so any leverage you want to make on it has to be multiplied by that small probability of it happening.

--

BTW, I'm curious to hear more about the mechanics of your scenario. The AGI hacks itself onto every (Internet-connected) computer in the world. Then what? Presumably this wouldn't cause extinction, just a lot of chaos and maybe years' worth of setback to the economy? Maybe it would increase chances of nuclear war, especially if the AGI could infect nuclear-warhead-related computer systems.

This could be an example of the non-extinction-level AGI disasters that I was referring to. Let me know if there are more ways in which it might cause total extinction, though.

Comment author: hyporational 07 April 2013 10:40:24PM 7 points [-]

one more voice for safety measures against extinction in a world where reducing extinction risk is hard and almost everyone has some incentives to invest in the issue.

Who are these almost everyone, what are they currently working on and what is their budget?

Comment author: Utilitarian 08 April 2013 10:50:09AM *  5 points [-]

This is a good point. :) I added an additional objection to the piece.

As an empirical matter, extinction risk isn't being funded as much as you suggest it should be if almost everyone has some incentives to invest in the issue.

There's a lot of "extinction risk" work that's not necessarily labeled as such: Biosecurity, anti-nuclear proliferation, general efforts to prevent international hostility by nation states, general efforts to reduce violence in society and alleviate mental illnesses, etc. We don't necessarily see huge investments in AI safety yet, but this will probably change in time, as we begin to see more AIs that get out of control and cause problems on a local scale. 99+% of catastrophic risks are not extinction risks, so as the catastrophes begin happening and affecting more people, governments will invest more in safeguards than they do now. The same can be said for nanotech.

In any event, even if budgets for extinction-risk reduction are pretty low, you also have to look at how much money can buy. Reducing risks is inherently difficult, because so much is out of our hands. It seems relatively easier to win over hearts and minds to utilitronium (especially at the margin right now, by collecting the low-hanging fruit of people who could be persuaded but aren't yet). And because so few people are pushing for utilitronium, it seems far easier to achieve a 1% increase in support for utilitronium than a 1% decrease in the likelihood of extinction.

Comment author: lukeprog 07 April 2013 08:30:02PM 1 point [-]

Relevant only to some parts of Tomasik's essay:

Ord, Why I'm Not a Negative Utilitarian

Comment author: Utilitarian 08 April 2013 10:48:22AM 4 points [-]

Thanks, Luke. See also this follow-up discussion to Ord's essay.

As you suggest with your "some" qualifier, my essay that benthamite shared doesn't make any assumptions about negative utilitarianism. I merely inserted parentheticals about my own views into it to avoid giving the impression that I'm personally a positive-leaning utilitarian.

Comment author: Jabberslythe 28 January 2013 08:58:33AM *  2 points [-]

You could reduce human suffering to 0 by reducing the number of humans to 0, so there's got to be another value greater than reducing suffering.

Almost all hedonistic utilitarians are concerned with maximizing happiness as well as minimizing suffering, including Brian. The reason that he talks about suffering so much is because, it is most people rank a unit of suffering as, say a -3 experience and a unit of suffering as, say, a -1 experience. And he thinks that there is much more suffering than happiness in the world and that it easier to prevent it.

(Sorry if I got any of this wrong Brian)

Comment author: Utilitarian 31 January 2013 06:30:21AM 0 points [-]

Thanks, Jabberslythe! You got it mostly correct. :)

The one thing I would add is that I personally think people don't usually take suffering seriously enough -- at least not really severe suffering like torture or being eaten alive. Indeed, many people may never have experienced something that bad. So I put high importance on preventing experiences like these relative to other things.

Comment author: Mestroyer 28 January 2013 12:57:57PM 2 points [-]

Another thing to worry about with CEV is that the nonperson predicates that whoever writes it decides on will cover things that you consider people, or would not like to see be destroyed at the end of an instrumental simulation.

Humans probably have no built-in intuitions for the details of distinction of things that deserve ethical consideration at the precision required for a nonperson predicate that can flag things as nonpersons that will be useful for instrumental simulations, and yet not flag a fully-detailed simulation of you or me as a nonperson. We don't have detailed enough introspection to know what "sentience" (whatever that means) is at a mechanical level. How can we care about the arrangement of parts that make "sentience," when we don't know what that arrangement is?

I think the process by which some people come to care about animals and others do not is probably highly dependent on which thought experiments they considered in which order, which label they first used for the category in their mind of "things that shouldn't be hurt."

The most memorable occasion when my person predicate changed was when I used to think that people could only exist in a basement universe. Simulations were automatically nonpersons. I thought to myself "if they aren't real I don't care." What changed my mind was the thought "If you ran a simulated version of me, and informed it that it was in a simulation, would it stop simulatedly caring about itself?" (The answer was no). But what if I had read LessWrong first, and become accustomed to thinking of myself as an insane (objectively speaking, not by human standards) biased ape, and said "No, but that's because I'm only human and sometimes have feelings that are contrary to my true ideal utility function. The simulated version may not alieve that he was not real, but he really wouldn't be real, so he Ideal_Mestroyer::should stop caring about himself." That thought isn't factually incorrect. If I had thought it back then, I might still care about "realness" in the same sense. But thinking about it now, it is too late, my terminal values have already changed, perhaps because of a misstep in my reasoning back then, and I am glad they have. But maybe the introduction of "real" (being directly made of physics and not in a simulation) as an important factor was originally based on mistaken reasoning too.

I think most of the features of our nonperson predicates are decided in the same way, partly randomly, based on reasoning mistakes and thought experiments considered first, (more randomly the more philosophical the person is), and partly through absorption from family and peers, which means it doesn't make sense for there to be a coherent extrapolated nonperson predicate for humanity (though you can still superpose a bunch of different ones).

Even if you don't really care about animals, your "person" category (or just "I care about this being" category) might be broader than SIAI's, and if it is, you should be afraid that vast numbers of people will be killed by terminating instrumental simulations.

Even so, if your person predicate is part of the CEV of humanity, perhaps an FAI could self-modify (after running some number of simulations using the old one that wasn't really that big compared to the number of people that would exist in a post-friendly-foom world)

So those people might not be that important to you, compared to what else is at stake. But if your nonperson predicate is not in humanity's CEV, and is uncommon enough that it's not worth it to humanity to accommodate you, and you disvalue death (and not just suffering) CEV might cause you to spend billions of years screaming.

Comment author: Utilitarian 29 January 2013 09:22:05AM *  1 point [-]

Interesting story. Yes, I think our intuitions about what kinds of computations we want to care about are easily bent and twisted depending on the situation at hand. In analogy with Dennett's "intentional stance," humans have a "compassionate stance" that we apply to some physical operations and don't apply to others. It's not too hard to manipulate these intuitions by thought experiments. So, yes, I do fear that other people may differ (perhaps quite a bit) in their views about what kinds of computations are suffering that we should avoid.

Comment author: Mestroyer 28 January 2013 11:53:17AM *  5 points [-]

I don't think that people valuing eternal torture of other humans is much of a concern, because they don't value it nearly as much as the people in question disvalue being tortured. I bet there are a lot more people who care about animals' feelings and who care a lot more, than those who care about the aesthetics of brutality in nature. I think the majority of people have more instincts of concern for animals than their actions suggest, because now it is convenient to screw over animals as an externality of eating tasty food, and the animals suffering are out of sight, and the connection between buying meat and animals living terrible lives elsewhere is hard to visualize. The same population that buys meat from farms that treat animals to awful lives also enacts animal cruelty laws. As evidence that caring more about animals is something that would be strengthened by thinking more about, consider the results of the [2012 LessWrong Survey]():

VEGETARIAN: No: 906, 76.6% Yes: 147, 12.4% No answer: 130, 11%

For comparison, 3.2% of US adults are vegetarian.

(Though some of this is probably due to lesswrongers being richer than the average american (something that's probably true because wealthy people more time to read about abstruse topics on the internet))

The biggest peculiarity of Brian Tomasik's utility function, that is least likely to ever be shared by the majority of humanity, is probably not that he cares about animals (even that he cares about insects) but that he cares so much more about suffering than happiness and other good things. (I am basing this assessment of his utility function on a post of his I read on http://www.felicifia.org a while ago, which I can't find now).

The exchange rate in your utility function between good things and bad things is pretty relevent to whether you should prefer CEV or paperclipping (And what the changes in the probabilities of each even based on actions you might take would have to be in order justify them) and whether you think lab universes would be a good thing.

So if you are not a negative utilitarian, or nearly one, even if Brian Tomasik's beliefs about matters of fact are very correlated with reality, be careful of his policy recommendations.

Comment author: Utilitarian 29 January 2013 09:12:50AM 4 points [-]

I bet there are a lot more people who care about animals' feelings and who care a lot more, than those who care about the aesthetics of brutality in nature.

Well, at the moment, there are hundreds of environmental-preservation organizations and basically no organizations dedicated to reducing wild-animal suffering. Environmentalism as a cause is much more mainstream than animal welfare. Just like the chickens that go into people's nuggets, animals suffering in nature "are out of sight, and the connection between [preserving pristine habitats] and animals living terrible lives elsewhere is hard to visualize."

It's encouraging that more LessWrongers are veg than average, although I think 12.4% is pretty typical for elite universities and the like as well. (But maybe that underscores your point.)

The biggest peculiarity of Brian Tomasik's utility function, that is least likely to ever be shared by the majority of humanity, is probably not that he cares about animals (even that he cares about insects) but that he cares so much more about suffering than happiness and other good things.

An example post. I care a lot about suffering, a little about happiness, and none about other things.

The exchange rate in your utility function between good things and bad things is pretty relevent to whether you should prefer CEV or paperclipping (And what the changes in the probabilities of each even based on actions you might take would have to be in order justify them) and whether you think lab universes would be a good thing.

Yep!

View more: Next