I think it's about a 0.75 probability, conditional upon smarter-than-human AI being developed. Guess I'm kind of an optimist. TL;DR I don't think it will be very difficult to impart your intentions into a sufficiently advanced machine.
I haven't seen any parts of Givewell's analyses that involve looking for the right buzzwords. Of course, it's possible that certain buzzwords subconsciously manipulate people at Givewell in certain ways, but the same can be said for any group, because every group has some sort of values.
Why do you expect that to be true?
Because they generally emphasize these values and practices when others don't, and because they are part of a common tribe.
How strongly? ("Ceteris paribus" could be consistent with an extremely weak effect.) Under what criterion for classifying people as EAs or non-EAs?
Somewhat weakly, but not extremely weakly. Obviously there is no single clear criteria, it's just about people's philosophical values and individual commitment. At most, I think that being a solid EA is about as important as having a couple...
I bet that most of the people who donated to Givewell's top charities were, for all intents and purposes, assuming their effectiveness in the first place. From the donor end, there were assumptions being made either way (and there must be; it's impractical to do all kinds of evaluation on one's own).
I think EA is something very distinct in itself. I do think that, ceteris paribus, it would be better to have a fund run by an EA than a fund not run by an EA. Firstly, I have a greater expectation for EAs to trust each other, engage in moral trades, be rational and charitable about each other's points of view, and maintain civil and constructive dialogue than I do for other people. And secondly, EA simply has the right values. It's a good culture to spread, which involves more individual responsibility and more philosophical clarity. Right now it's embryo...
It seems to me that Givewell has already acknowledged perfectly well that VillageReach is not a top effective charity. It also seems to me that there's lots of reasons one might take GiveWell's recommendations seriously, and that getting "particularly horrified" about their decision not to research exactly how much impact their wrong choice didn't have is a rather poor way to conduct any sort of inquiry on the accuracy of organizations' decisions.
It was very much not obvious to me that GiveWell doubted its original VillageReach recommendation until I emailed. What published information made this obvious to you?
The main explanation I could find for taking VillageReach off the Top Charities list was that they no longer had room for more funding. At the time I figured this simply meant they'd finished scaling up inside the country and didn't have more work to do of the kind that earned the Top Charity recommendation.
In fact, it seems to me that the less intelligent an organism is, the easier its behavior can be approximated with model that has a utility function!
Only because those organisms have fewer behaviors in general. If you put a human in an environment where its options and sensory inputs were as simple as those experienced by apes and cats, humans would probably look like equally simple utility maximizers.
Kantian ethics: do not violate the categorical imperative. It's derived logically from the status of humans as rational autonomous moral agents. It leads to a society where people's rights and interests are respected.
Utilitarianism: maximize utility. It's derived logically from the goodness of pleasure and the badness of pain. It leads to a society where people suffer little and are very happy.
Virtue ethics: be a virtuous person. It's derived logically from the nature of the human being. It leads to a society where people act in accordance with moral ideals.
Etc.
pigs strike a balance between the lower suffering, higher ecological impact of beef and the higher suffering, lower ecological impact of chicken.
This was my thinking for coming to the same conclusion. But I am not confident in it. Just because something minimaxes between two criteria doesn't mean that it minimizes overall expected harm.
All of the architectures assumed by people who promote these scenarios have a core set of fundamental weaknesses (spelled out in my 2014 AAAI Spring Symposium paper).
The idea of superintelligence at stake isn't "good at inferring what people want and then decides to do what people want," it's "competent at changing the environment". And if you program an explicit definition of 'happiness' into a machine, its definition of what it wants - human happiness - is not going to change no matter how competent it becomes. And there is no reas...
I came here to write exactly what gjm said, and your response is only to repeat the assertion "Scenarios in which the AI Danger comes from an AGI that is assumed to be an RL system are so ubiquitous that it is almost impossible to find a scenario that does not, when push comes to shove, make that assumption."
What? What about all the scenarios in IEM or Superintelligence? Omohundro's paper on instrumental drives? I can't think of anything which even mentions RL, and I can't see how any of it relies upon such an assumption.
So you're alleging that d...
In Bostrom's dissertation he says it's not clear if number of observers or the number of observer-moments is the appropriate reference class for anthropic reasoning.
I don't see how you are jumping to the fourth disjunct though. Like, maybe they run lots of simulations which are very short? But surely they would run enough to outweigh humanity's real history whichever way you measure it. Assuming they have posthuman levels of computational power.
In other words, a decision theory, complete with an algorithm (so you can actually use it), and a full set of terminal goals. Not what anyone else means by "moral theory'.
When people talk about moral theories they refer to systems which describe the way that one ought to act or the type of person that one ought to be. Sure, some moral theories can be called "a decision theory, complete with an algorithm (so you can actually use it), and a full set of terminal goals," but I don't see how that changes anything about the definition of a moral theory.
To say that you may chose any one of two actions when it doesn’t matter which one you chose since they have the same value, isn’t to give “no guidance”.
Proves my point. That's no different from how most most moral theories respond to questions like "which shirt do I wear". So this 'completeness criterion' has to be made so weak as to be uninteresting.
Among hedonistic utilitarians it's quite normal to demand both completeness
Utilitarianism provides no guidance on many decisions: any decision where both actions produce the same utility.
Even if it is a complete theory, I don't think that completeness is demanded of the theory; rather it's merely a tenet of it. I can't think of any good a priori reasons to expect a theory to be complete in the first place.
The question needs to cover how one should act in all situations, simply because we want to answer the question. Otherwise we’re left without guidance and with uncertainty.
Well first, we normally don't think of questions like which clothes to wear as being moral. Secondly, we're not left without guidance when morality leaves these issues alone: we have pragmatic reasons, for instance. Thirdly, we will always have to deal with uncertainty due to empirical uncertainty, so it must be acceptable anyway.
...There is one additional issue I would like to highli
See Omohundro's paper on convergent instrumental drives
It seems like hedging is the sort of thing which tends to make the writer sound more educated and intelligent, if possibly more pretentious.
It's unjustified in the same way that vilalism was an unjustified explanation of life: it's purely a product of our ignorance.
It's not. Suppose that the ignorance went away: a complete physical explanation of each of our qualia - "the redness of red comes from these neurons in this part of the brain, the sound of birds flapping their wings is determined by the structure of electric signals in this region," and so on - would do nothing to remove our intuitions about consciousness. But a complete mechanistic explanation of how organ systems wor...
You should take a look at the last comment he made in reply to me, where he explicitly ascribed to me and then attacked (at length) a claim which I clearly stated that I didn't hold in the parent comment. It's amazing how difficult it is for the naive-eliminativist crowd to express cogent arguments or understand the positions which they attack, and a common pattern I've noticed across this forum as well as others.
Not too long ago, it would also have been quite easy to conceive of a world in which heat and motion were two separate things. Today, this is no longer conceivable.
But it is conceivable for thermodynamics to be caused by molecular motion. No part of that is (or ever was, really) inconceivable. It is inconceivable for the sense qualia of heat to be reducible to motion, but that's just another reason to believe that physicalism is wrong. The blog post you linked doesn't actually address the idea of inconceivability.
...If something seems conceivable to you
I claim that it is "conceivable" for there to be a universe whose psychophysical laws are such that only the collection of physical states comprising my brainstates are conscious, and the rest of you are all p-zombies.
Yes. I agree that it is conceivable.
...Now then: I claim that by sheer miraculous coincidence, this universe that we are living in possesses the exact psychophysical laws described above (even though there is no way for my body typing this right now to know that), and hence I am the only one in the universe who actually experienc
Well, first off, I personally think the Zombie World is logically impossible, since I treat consciousness as an emergent phenomenon rather than a mysterious epiphenomenal substance; in other words, I reject the argument's premise: that the Zombie World's existence is "conceivable".
And yet it seems really quite easy to conceive of a p zombie. Merely claiming that consciousness is emergent doesn't change our ability to imagine the presence or absence of the phenomenon.
...That being said, if you do accept the Zombie World argument, then there's no
4 is not a correct summary because consciousness being extra physical doesn't imply epiphenominalism; the argument is specifically against physicalism, so it leaves other forms of dualism and panpsychism on the table.
5 and onwards is not correct, Chalmers does not believe that. Consciousness being nonphysical does not imply a lack of knowledge of it, even if our experience of consciousness is not causally efficacious (though again I note that the p zombie argument doesn't show that consciousness is not causally efficacious, Chalmers just happens to believe...
Which seems to suggest that epiphenominalism either begs the question,
Well, they do have arguments for their positions.
or multiplies entities unnecessarily by accepting unjustified intuitions.
It actually seems very intuitive to most people that subjective qualia are different from neurophysical responses. It is the key issue at stake with zombie and knowledge arguments and has made life extremely difficult for physicalists. I'm not sure in what way it's unjustified for me to have an intuition that qualia are different from physical structures, and r...
In what ways, and for what reasons, did people think that cybersecurity had failed?
Mostly that it's just so hard to keep things secure. Organizations have been trying for decades to ensure security but there are continuous failures and exploits. One person mentioned that one third of exploits take advantage of security systems themselves.
What techniques from cybersecurity were thought to be relevant?
Don't really remember any specifics, but I think formal methods were part of it.
...Any idea what Mallah meant by “non-self-centered ontologies”? I am ima
Flavor is distinctly a phenomenal property and a type of qualia.
It is metaphysically impossible for distinctly physical properties to differ between two objects which are physically identical. We can't properly conceive of a cookie that is physically identical to an Oreo yet contains different chemicals, is more massive or possessive of locomotive powers. Somewhere in our mental model of such an item, there is a contradiction.
Chalmers does believe that consciousness is a direct product of physical states. The dispute is about whether consciousness is identical to physical states.
Chalmers does not believe that p-zombies are possible in the sense that you could make one in the universe. He only believes it's possible that under a different set of psychophysical laws, they could exist.
Yes, this is called qualia inversion and is another common argument against physicalism. There's a detailed discussion of it here: http://plato.stanford.edu/entries/qualia-inverted/
Unlike the other points which I raised above, this one is semantic. When we talk about "knowledge," we are talking about neurophysical responses, or we are talking about subjective qualia, or we are implicitly combining the two together. Epiphenomenalists, like physicalists, believe that sensory data causes the neurophysical responses in the brain which we identify with knowledge. They disagree with physicalists because they say that our subjective qualia are epiphenomenal shadows of those neurophysical responses, rather than being identical to them. There is no real world example that would prove or disprove this theory because it is a philosophical dispute. One of the main arguments for it is, well, the zombie argument.
is why or if the p-zombie philosopher postulate that other persons have consciousness.
Because consciousness supervenes upon physical states, and other brains have similar physical states.
This argument is not going to win over their heads and hearts. It's clearly written for a reductionist reader, who accepts concepts such as Occam's Razor and knowing-what-a-correct-theory-looks-like.
I would suggest that people who have already studied this issue in depth would have other reasons for rejecting the above blog post. However, you are right that philosophers in general don't use Occam's Razor as a common tool and they don't seem to make assumptions about what a correct theory "looks like."
...If conceivability does not imply logical
I don't believe that I experience qualia.
Wait, what?
3 doesn't follow from 2, it follows from a contradiction between 1+2.
Well, first of all, 3 isn't a statement, it's saying "consider a world where..." and then asking a question about whether philosophers would talk about consciousness. So I'm not sure what you mean by suggesting that it follows or that it is true.
1 and 2 are not contradictions. Conversely, 1 and 2 are basically saying the exact same thing.
...1 states that consciousness has no effect upon matter, and yet it's clear from observation that the concept of subjectivity only follows i
Indeed. The condensed argument against p-zombies:
I would hope not. 3 is entirely conceivable if we grant 2, so 4 is unsupported, and nothing that EY said supports 4. 5 does not follow from 3 or 4, though it's bundled up in the definition of a p-zombie and follows from 1 and 2 anyway. In any case, 6 does not follow from 5.
What EY is saying is that it's highly implausible for all of our ideas and talk of consciousness to have come to be if subjective consciousness does not play a causal role in our thinking.
...Except such discussions would have no motivati
Well that's answered by what I said about psychophysical laws and the evolutionary origins of consciousness. What caused us to believe in consciousness is not (necessarily) the same issue as what reasons we have to believe it.
This was longer than it needed to be, and in my opinion, somewhat mistaken.
The zombie argument is not an argument for epiphenomenalism, it's an argument against physicalism. It doesn't assume that interactionist dualism is false, regardless of the fact that Chalmers happens to be an epiphenomenalist.
Chalmers furthermore specifies that this true stuff of consciousness is epiphenomenal, without causal potency—but why say that?
Maybe because interactionism violates the laws of physics and is somewhat at odds with everything we (think we) know about cogniti...
In fairness, I didn't directly ask any of them about it, and it wasn't really discussed. There could have been some who had read the relevant work, and many who believed it to be reasonable, but just didn't happen to speak up during the presentations or in any of the conversations I was in.
There is no objective absolute morality that exists in a vacuum.
No, that's highly contentious, and even if it's true, it doesn't grant a license to promote any odd utility rule as ideal. The anti-realist also may have reason to prefer a simpler version of morality.
...Utility theory, prisoner's dilemma, Occam's razor, and many other mathematical structures put constraints on what a self-consistent, formalized morality has to be like. But they can't and won't pinpoint a single formula in the huge hypothesis space of morality, but we'll always have to rely
Would you accept a lottery where there was 1 ticket to maintain your life as a satisfied cookie utility monster and hundreds of trillions of tickets to become a miserable enslaved cookie maker?
Or, after rational reflection and experiencing the alternate possibilities, would you rather prefer a guaranteed life of threshold satisfaction?
The problem is that by doing that you are making your position that much more arbitrary and contrived. It would be better if we could find a moral theory that has solid parsimonious basis, and it would be surprising if the fabric of morality involved complicated formulas.
Thanks. I will give some of those articles a look when I have the chance. However, it isn't true that every activity is competitive in nature. Many projects are cooperative, in which case it's not necessarily a problem if you and other people are taking similar approaches and doing them well. We also shouldn't overestimate the competition and assume that they are going to be applying probabilistic reasoning, when in reality we can still outperform by applying basic rules of rationality.
So for us to understand what you're even trying to say, you want us to read a bunch of articles, talk to one of your friends, listen to a speech, and only then will we become EAs good enough for you? No thanks.
This is very old but I just wanted to say that I am basically considering changing my college choice due to finding out about this research. Thanks so much for putting this post up and spreading awareness.
Maybe I am unfamiliar with the specifics of simulated reality. But I don't understand how it is assumed (or even probable, given Occam's Razor) that if we are simulated then there are copies of us. What is implausible about the possibility that I'm in a simulation and I'm the only instance of me that exists?
Sorry if this has topic has been beaten to death already here. I was wondering if anyone here has seen this paper and has an opinion on it.
The abstract: "This paper argues that at least one of the following propositions is true: (1) the human species is very likely to go extinct before reaching a “posthuman” stage; (2) any posthuman civilization is extremely unlikely to run a significant number of simulations of their evolutionary history (or variations thereof); (3) we are almost certainly living in a computer simulation. It follows that the belief t...
Hi, I've been intermittently lurking here since I started reading HPMOR. So now I joined and the first thing I wanted to bring up is this paper which I read about the possibility that we are living in a simulation. The abstract:
"This paper argues that at least one of the following propositions is true: (1) the human species is very likely to go extinct before reaching a “posthuman” stage; (2) any posthuman civilization is extremely unlikely to run a significant number of simulations of their evolutionary history (or variations thereof); (3) we are al...
I find myself to have a much clearer and cooler head when it comes to philosophy and debate around the subject. Previously I had a really hard time squaring utilitarianism with the teachings of religion, and I ended up being a total heretic. Now I feel like everything makes sense in a simpler way.
1%? Shouldn't your basic uncertainty over models and paradigms be great enough to increase that substantially?