Jonah, I agree with what you say at least in principle, even if you would claim I don't follow it in practice. A big advantage of being Bayesian is that you retain probability mass on all the options rather than picking just one. (I recall many times being dismayed with hacky approximations like MAP that let you get rid of the less likely options. Similarly when people conflate the Solomonoff probability of a bitstring with the shortest program that outputs it, even though I guess in that case, the shortest program necessarily has at least as much probability as all the others can combined.)

My main comment on your post is that it's hard to keep track of all of these things computationally. Probably you should try, but it can get messy. It's also possible that in keeping track of too many details, you introduce more errors than if you had kept the analysis simple. On many questions in physics, ecology, etc., there's a single factor that dominates all the rest. Maybe this is less true in human domains because rational agents tend to produce efficiencies due to eating up the free lunches.

So, I'm in favor of this approach if you can do it and make it work, but don't let the best be the enemy of the good. Focus on the strong arguments first, and only if you have the bandwidth go on to think about the weak ones too.

As far as we can tell these new vegetarians were eating like normal Americans before they saw the videos.

I suspect they were not. People who have a more entrenched habit requiring greater life alteration to change it are less likely to give it up.

Of the people I've known who went vegetarian, I don't think any of them went from being big meat eaters to totally abstaining in a single step.

However, if the AGI is hiding itself, it could go a long way before people realized what was going on.

Exactly. On the one hand the AGI doesn't try to let humans get wind of it's plans. On the other hand it's going to produce distractions.

You have to remember how delusional some folks are. Imaging trying to convince the North Korean's that they have to destroy their computers because those computer are infested with an evil AI.

Even in the US nearly half of the population still believes in creationism. How many of them can be convinced that the evil government is trying to take away their computers to establish a dictatorship?

Before the government goes attempts to trash the computer the AI sent an email to a conspiracy theory website, where it starts revealing some classified documents it aquired through hacking that show government misbehavior.

Then it sents an email to the same group saying that the US government is going to shut down all civilian computers because freedom of speech is to dangerous to the US government and that the US government will be using the excuse that the computers are part of a Chinese botnet.

In our time you need computers to stock supermarket shelves with goods. Container ships need GPS and see charts to navigate.

People start fighting each other. Some are likely to blame the people who wanted to thrash the computers as responsible for the mess.

Even if you can imagine shutting of all computer in 2013, in 2033 most cars will be computers in which the AI can rest. A lot of military firepower will be in drones that the AI can control.

I think he meant more along the lines of computers/robots/ non-super AIs becoming more powerful, IDK.

We don't necessarily see huge investments in AI safety yet, but this will probably change in time, as we begin to see more AIs that get out of control and cause problems on a local scale.

Once we see an out of control AI it's to late to do AI safety. Given current computer security the AI could hack itself into every computer in the world and resist easy shutdown.

When it comes to low probability high impact events waiting for small problem to cause awareness of the issue is just dangerous.

one more voice for safety measures against extinction in a world where reducing extinction risk is hard and almost everyone has some incentives to invest in the issue.

Who are these almost everyone, what are they currently working on and what is their budget?

Relevant only to some parts of Tomasik's essay:

Ord, Why I'm Not a Negative Utilitarian

You could reduce human suffering to 0 by reducing the number of humans to 0, so there's got to be another value greater than reducing suffering.

Almost all hedonistic utilitarians are concerned with maximizing happiness as well as minimizing suffering, including Brian. The reason that he talks about suffering so much is because, it is most people rank a unit of suffering as, say a -3 experience and a unit of suffering as, say, a -1 experience. And he thinks that there is much more suffering than happiness in the world and that it easier to prevent it.

(Sorry if I got any of this wrong Brian)

Another thing to worry about with CEV is that the nonperson predicates that whoever writes it decides on will cover things that you consider people, or would not like to see be destroyed at the end of an instrumental simulation.

Humans probably have no built-in intuitions for the details of distinction of things that deserve ethical consideration at the precision required for a nonperson predicate that can flag things as nonpersons that will be useful for instrumental simulations, and yet not flag a fully-detailed simulation of you or me as a nonperson. We don't have detailed enough introspection to know what "sentience" (whatever that means) is at a mechanical level. How can we care about the arrangement of parts that make "sentience," when we don't know what that arrangement is?

I think the process by which some people come to care about animals and others do not is probably highly dependent on which thought experiments they considered in which order, which label they first used for the category in their mind of "things that shouldn't be hurt."

The most memorable occasion when my person predicate changed was when I used to think that people could only exist in a basement universe. Simulations were automatically nonpersons. I thought to myself "if they aren't real I don't care." What changed my mind was the thought "If you ran a simulated version of me, and informed it that it was in a simulation, would it stop simulatedly caring about itself?" (The answer was no). But what if I had read LessWrong first, and become accustomed to thinking of myself as an insane (objectively speaking, not by human standards) biased ape, and said "No, but that's because I'm only human and sometimes have feelings that are contrary to my true ideal utility function. The simulated version may not alieve that he was not real, but he really wouldn't be real, so he Ideal_Mestroyer::should stop caring about himself." That thought isn't factually incorrect. If I had thought it back then, I might still care about "realness" in the same sense. But thinking about it now, it is too late, my terminal values have already changed, perhaps because of a misstep in my reasoning back then, and I am glad they have. But maybe the introduction of "real" (being directly made of physics and not in a simulation) as an important factor was originally based on mistaken reasoning too.

I think most of the features of our nonperson predicates are decided in the same way, partly randomly, based on reasoning mistakes and thought experiments considered first, (more randomly the more philosophical the person is), and partly through absorption from family and peers, which means it doesn't make sense for there to be a coherent extrapolated nonperson predicate for humanity (though you can still superpose a bunch of different ones).

Even if you don't really care about animals, your "person" category (or just "I care about this being" category) might be broader than SIAI's, and if it is, you should be afraid that vast numbers of people will be killed by terminating instrumental simulations.

Even so, if your person predicate is part of the CEV of humanity, perhaps an FAI could self-modify (after running some number of simulations using the old one that wasn't really that big compared to the number of people that would exist in a post-friendly-foom world)

So those people might not be that important to you, compared to what else is at stake. But if your nonperson predicate is not in humanity's CEV, and is uncommon enough that it's not worth it to humanity to accommodate you, and you disvalue death (and not just suffering) CEV might cause you to spend billions of years screaming.

I don't think that people valuing eternal torture of other humans is much of a concern, because they don't value it nearly as much as the people in question disvalue being tortured. I bet there are a lot more people who care about animals' feelings and who care a lot more, than those who care about the aesthetics of brutality in nature. I think the majority of people have more instincts of concern for animals than their actions suggest, because now it is convenient to screw over animals as an externality of eating tasty food, and the animals suffering are out of sight, and the connection between buying meat and animals living terrible lives elsewhere is hard to visualize. The same population that buys meat from farms that treat animals to awful lives also enacts animal cruelty laws. As evidence that caring more about animals is something that would be strengthened by thinking more about, consider the results of the [2012 LessWrong Survey]():

VEGETARIAN: No: 906, 76.6% Yes: 147, 12.4% No answer: 130, 11%

For comparison, 3.2% of US adults are vegetarian.

(Though some of this is probably due to lesswrongers being richer than the average american (something that's probably true because wealthy people more time to read about abstruse topics on the internet))

The biggest peculiarity of Brian Tomasik's utility function, that is least likely to ever be shared by the majority of humanity, is probably not that he cares about animals (even that he cares about insects) but that he cares so much more about suffering than happiness and other good things. (I am basing this assessment of his utility function on a post of his I read on http://www.felicifia.org a while ago, which I can't find now).

The exchange rate in your utility function between good things and bad things is pretty relevent to whether you should prefer CEV or paperclipping (And what the changes in the probabilities of each even based on actions you might take would have to be in order justify them) and whether you think lab universes would be a good thing.

So if you are not a negative utilitarian, or nearly one, even if Brian Tomasik's beliefs about matters of fact are very correlated with reality, be careful of his policy recommendations.