All of HopeFox's Comments + Replies

HopeFox00

I agree, intuition is very difficult here. In this specific scenario, I'd lean towards saying yes - it's the same person with a physically different body and brain, so I'd like to think that there is some continuity of the "person" in that situation. My brain isn't made of the "same atoms" it was when I was born, after all. So I'd say yes. In fact, in practice, I would definitely assume said robot and software to have moral value, even if I wasn't 100% sure.

However, if the original brain and body weren't destroyed, and we now had two ap... (read more)

0hairyfigment
Have you read the quantum physics sequence? Would you agree with me that nothing you learn about seemingly unrelated topics like QM should have the power to destroy the whole basis of your morality?
5MixedNuts
Yes, the consensus seems to be that running two copies of yourself in parallel doesn't give you more measure or moral weight. But if the copies receive diferent inputs, they'll eventually (frantic handwaving) diverge into two different people who both matter. (Maybe when we can't retrieve Copy-A's current state from Copy-B's current state and the respective inputs, because information about the initial state has been destroyed?)
HopeFox-10

What issues does your best atheist theory have?

My biggest problem right now is all the stuff about zombies, and how that implies that, in the absence of some kind of soul, a computer program or other entity that is capable of the same reasoning processes as a person, is morally equivalent to a person. I agree with every step of the logic (I think, it's been a while since I last read the sequence), but I end up applying it in the other direction. I don't think a computer program can have any moral value, therefore, without the presence of a soul, people ... (read more)

4Oscar_Cunningham
In the spirit of your (excellent) new post, I'll attack all the weak points of your argument at once: * You define "soul" as: This definition doesn't give souls any of their normal properties, like being the seat of subjective experience, or allowing free will, or surviving bodily death. That's fine, but we need to be on the look-out in case these meanings sneak in as connotations later on. (In particular, the "Zombies" sequence doesn't talk about moral worth, but does talk about subjective experience, so its application here isn't straight forward. Do you believe that a simulation of a human would have subjective experience?) * "Souls" don't provide any change in anticipation. You haven't provided any mechanism by which other people having souls causes me to think that those other people have moral worth. Furthermore it seems that my belief that others have moral worth can be fully explained by my genes and my upbringing. * You haven't stated any evidence for the claim that computer programs can't have moral value, and this isn't intuitively obvious to me. * You've produced a dichotomy between two very unlikely hypotheses. I think the correct answer in this case isn't to believe the least unlikely hypothesis, but is instead to assume that the answer is some third option you haven't thought of yet. For instance you could say "I withhold judgement on the existence of souls and the nature of moral worth until I understand the nature of subjective experience". * The existence of souls as you've defined them doesn't imply theism. Not even slightly. (EDIT: Your argument goes: 'By the "Zombies" sequence, simulations are concious. By assumption, simulations have no moral worth. Therefore concious does not imply moral worth. Call whatever does imply moral worth a soul. Souls exist, therefore theism.' The jump between the penultimate and the ultimate step is entirely powered by connotations of the word "soul", and is therefore invalid.) Also you say this: (I'm s
3Vladimir_Nesov
You don't need an additional ontological entity to reflect a judgment (and judgments can differ between different people or agents). You don't need special angry atoms to form an angry person, that property can be either in the pattern of how the atoms are arranged, or in the way you perceive their arrangement. See these posts: * http://lesswrong.com/lw/oi/mind_projection_fallacy/ * http://lesswrong.com/lw/oj/probability_is_in_the_mind/ * http://lesswrong.com/lw/ro/2place_and_1place_words/ * http://lesswrong.com/lw/oo/explaining_vs_explaining_away/ * http://lesswrong.com/lw/p3/angry_atoms/
0Laoch
I'd love to know why moral value => presence of a soul? Also theist is a very vague term taken by itself could mean anything. Care to enlighten us?
2Kaj_Sotala
Thanks. Can you be more specific about what you mean by a soul? To me, it sounds like you're just using it as a designation of something that has moral value to you. But that doesn't need to imply anything supernatural; it's just an axiom in your moral system.
3jimrandomh
It's hard to build intuitions about the moral value of intelligent programs right now, because there aren't any around to talk to. But consider a hypothetical that's as close to human as possible: uploads. Suppose someone you knew decided to undergo a procedure where his brain would be scanned and destroyed, and then a program based on that scan was installed on a humanoid robot body, so that it would act and think like he did; and when you talked to the robot, he told you that he still felt like the same person. Would that robot and the software on it have moral value?
HopeFox20

It's about how, if you're attacking somebody's argument, you should attack all of the bad points of it simultaneously, so that it doesn't look like you're attacking one and implicitly accepting the others. With any luck, it'll be up tonight.

HopeFox70

Hi, I've been lurking on Less Wrong for a few months now, making a few comments here and there, but never got around to introducing myself. Since I'm planning out an actual post at the moment, I figured I should tell people where I'm coming from.

I'm a male 30-year-old optical engineer in Sydney, Australia. I grew up in a very scientific family and have pretty much always assumed I had a scientific career ahead of me, and after a couple of false starts, it's happened and I couldn't ask for a better job.

Like many people, I came to Less Wrong from TVTropes vi... (read more)

0Oscar_Cunningham
Welcome! Exciting! What's it about?
1Kaj_Sotala
Welcome! What issues does your best atheist theory have?
HopeFox-20

Assuming rational agents with a reasonable level of altruism (by which I mean, incorporating the needs of other people and future generations into their own utility functions, to a similar degree to what we consider "decent people" to do today)...

If such a person figures that getting rid of the Nazis or the Daleks or whoever the threat of the day is, is worth a tiny risk of bringing about the end of the world, and their reasoning is completely rational and valid and altrustic (I won't say "unselfish" for reasons discussed elsewhere in t... (read more)

0PhilGoetz
You've sidestepped my argument, which is that just the existential risks that are worth it are enough to guarantee destroying the universe in a cosmologically short time.
HopeFox10

Thinking about this in commonsense terms is misleading, because we can't imagine the difference between 8x utility and 16x utility

I can't even imagine doubling my utility once, if we're only talking about selfish preferences. If I understand vNM utility correctly, then a doubling of my personal utility is a situation which I'd be willing to accept a 50% chance of death in order to achieve (assuming that my utility is scaled so that U(dead) = 0, and without setting a constant level, we can't talk about doubling utility). Given my life at the moment (apar... (read more)

HopeFox60

"Every time you draw a card with a star, I'll double your utility for the rest of your life. If you draw a card with a skull, I'll kill you."

Sorry if this question has already been answered (I've read the comments but probably didn't catch all of it), but...

I have a problem with "double your utility for the rest of your life". Are we talking about utilons per second? Or do you mean "double the utility of your life", or just "double your utility"? How does dying a couple of minutes later affect your utility? Do you... (read more)

HopeFox00

Perfect decision-makers, with perfect information, should always be able to take the optimal outcome in any situation. Likewise, perfect decision-makers with limited information should always be able to choose the outcome with the best expected payoff under strict Bayesian reasoning.

However, when the actor's decision-making process becomes part of the situation under consideration, as happens when Katemega scrutinises Joe's potential for leaving her in the future, then the perfect decision-maker is only able to choose the optimal outcome if he is also capa... (read more)

1Bongo
I have to note that an agent using one of the new decision theories sometimes discussed around here, like UDT, wouldn't leave Katemega and wouldn't need self-modification or precommitment to not leave her.
HopeFox10

It's an interesting situation, and I can see the parallel to Newcombe's Problem. I'm not certain that it's possible for a person to self-modify to the extent that he will never leave his wife, ever, regardless of the very real (if small) doubts he has about the relationship right now. I don't think I could ever simultaneously sustain the thoughts "There's about a 10% chance that my marriage to my wife will make me very unhappy" and "I will never leave her no matter what". I could make the commitment financially - that, even if the marri... (read more)

HopeFox60

Talking with people that do not agree with you as though they were people. That is taking what they say seriously and trying to understand why they are saying what they say. Asking questions helps. Also, assume that they have reasons that seem rational to them for what they say or do, even if you disagree.

I think this is a very important point. If we can avoid seeing our political enemies as evil mutants, then hopefully we can avoid seeing our conversational opponents as irrational mutants. Even after discounting the possibility that you, personally, mi... (read more)

HopeFox20

I don't know how to port this strategy over to verbal acuity for rationality.

Perhaps by vocalising simple logic? When you make a simple decision, such as "I'm going to walk to work today instead of catching the bus", go over your logic for the decision, even after you've started walking, as if you're explaining your decision to someone else. I often do this (not out loud, but as a mental conversation), just for something to pass the time, and I find that it actually helps me organise my thoughts and explain my logic to other real people.

HopeFox170

Sexual Weirdtopia:

The government takes a substantial interest in people's sex lives. People are expected to register their sexual preferences with government agencies. A certain level of sexual education and satisfaction is presumed to be a basic right of humanity, along with health care and enough income to live on. Workers are entitled to five days' annual leave for seeking new or maintaining old romantic and sexual relationships, and if your lover leaves you because you're working too hard, you can sue your employer and are likely to win. Private prosti... (read more)

9NancyLebovitz
Pushing the weirdness farther: assume tech for adjusting orientation, level of desire, and desire for exclusiveness. Either there are no side effects, or the side effects are considered to be low compared to the effects of people not getting what they want sexually. Individual sexual parameters are adjusted to maximize sexual harmony in each person's social network (individual change is minimized-- this is presumably NP-hard), so that sexual parameter combinations change as they move through social networks. Consent is preserved for individual encounters, but being subject to having one's parameters adjusted is mandatory.
HopeFox110

I think I've started to do this already for Disputing Definitions, as has my girlfriend, just from listening to me discussing that article without reading it herself. So that's a win for rationality right there.

To take an example that comes up in our household surprisingly often, I'll let the disputed definition be " steampunk ". Statements of the form "X isn't really steampunk!" come up a lot on certain websites, and arguments over what does or doesn't count as steampunk can be pretty vicious. After reading "Disputing Definitions&... (read more)

HopeFox40

A person who can kill another person might well want 5$, for whatever reason. In contrast, a person who can use power from beyond the Matrix to torture 3^^^3 people already has IMMENSE power. Clearly such a person has all the money they want, and even more than that in the influence that money represents. They can probably create the money out of nothing. So already their claims don't make sense if taken at face value.

Ah, my mistake. You're arguing based on the intent of a legitimate mugger, rather than the fakes. Yes, that makes sense. If we let f(N) b... (read more)

HopeFox10

This is a very good point - the higher the number chosen, the more likely it is that the mugger is lying - but I don't think it quite solves the problem.

The probability that a person, out to make some money, will attempt a Pascal's Mugging can be no greater than 1, so let's imagine that it is 1. Every time I step out of my front door, I get mobbed by Pascal's Muggers. My mail box is full of Pascal's Chain Letters. Whenever I go online, I get popups saying "Click this link or 3^^^^3 people will die!". Let's say I get one Pascal-style threat every ... (read more)

-1Polymeron
But that is precisely it - it's no longer a Pascal mugging if the threat is credible. That is, in order to be successful, the mugger needs to be able to up the utility claim arbitrarily! It is assumed that we already know how to handle a credible threat, what we didn't know how to deal with was a mugger who could always make up a bigger number, to a degree where the seeming impossibility of the claim no longer offsets the claimed utility. But as I showed, this only works if you don't enter the mugger's thought process into the calculation. This actually brings up an important corollary to my earlier point: The higher the number, the less likely the coupling is between the mugger's claim and the mugger's intent. A person who can kill another person might well want 5$, for whatever reason. In contrast, a person who can use power from beyond the Matrix to torture 3^^^3 people already has IMMENSE power. Clearly such a person has all the money they want, and even more than that in the influence that money represents. They can probably create the money out of nothing. So already their claims don't make sense if taken at face value. Maybe the mugger just wants me to surrender to an arbitrary threat? But in that case, why me? If the mugger really has immense power, they could create a person they know would cave in to their demands. Maybe I'm special for some reason. But if the mugger is REALLY that powerful, wouldn't they be able to predict my actions beforehand, a-la Omega? Each rise in claimed utility brings with it a host of assumptions that need to be made for the action-claimed reaction link to be maintained. And remember, the mugger's ability is not the only thing dictating expected utility, but also the mugger's intentions. Each such assumption not only weakens the probability of the mugger carrying out their threat because they can't, it also raises the probability of the mugger rewarding refusal and/or punishing compliance. Just because the off-chance comes t
HopeFox00

This problem sounds awfully similar to the halting problem to me. If we can't tell whether a Turing machine will eventually terminate without actually running it, how could we ever tell if a Turing machine will experience consciousness without running it?

Has anyone attempted to prove the statement "Consciousness of a Turing machine is undecideable"? The proof (if it's true) might look a lot like the proof that the halting problem is undecideable. Sadly, I don't quite understand how that proof works either, so I can't use it as a basis for the con... (read more)

1VNKKET
Your conjecture seems to follow from Rice's theorem, assuming the personhood of a running computation is a property of the partial function its algorithm computes. Also, I think you can prove your conjecture by taking a certain proof that the Halting Problem is undecidable and replacing 'halts' with 'is conscious'. I can track this down if you're still interested. But this doesn't mess up Eliezer's plans at all: you can have "nonhalting predicates" that output "doesn't halt" or "I don't know", analogous to the nonperson predicates proposed here.
0hairyfigment
Um, I happened to write an explanation of the Halting Problem proof in a comment over here. Please tell me which parts seem unclear to you.
5orthonormal
The halting problem doesn't imply that we can never tell whether a particular program halts without actually running it. (You can think of many simple programs which definitely halt, and other simple programs which are definitely infinite loops.) It means, instead, that there exist relatively short but extremely pathological Turing machines, such that no Turing machine can be built that could solve the halting problem for every Turing machine. (Indeed, the idea of the proof is that a reputed halting-problem-solver is itself pathological, as can be seen by feeding it a modified version of itself as input.) But these pathological ones are not at all the kind of Turing machines we would create to do any functional task; the only reason I could think for us to seek them out would be to find Busy Beaver numbers.
HopeFox70

Do we even need the destination? When you consider "fun" as something that comes from a process, from the journey of approaching a goal, then wouldn't it make sense to disentangle the journey and the goal? We shouldn't need the destination in order to make the journey worthwhile. I mean, if the goal were actually important, then surely we'd just get our AI buddies to implement the goal, while I was off doing fun journey stuff.

For a more concrete example:

I like baking fruitcakes. (Something I don't do nearly often enough these days.) Mixing the ra... (read more)

5taryneast
I totally agree... there are heaps of processes that I enjoy far more than the actual end-result. Crochet is my example. I'm quite happy to continue crocheting something pretty (it has to be pretty - I don't enjoy crocheting abominations) for a long time and never "owning a crocheted thing" at the end. Before I hit upon the solution, I spent a long time starting projects - some of which I finished, but lots I didn't... because I didn't care about finishing - just about doing. Of course, couple this with an aversion to destroying something I've already made (which might have solved the problem by turning it into a sisyphean task). and I got a lot of "why don't you ever finish anything?" from my mother. The question usually comes as "why don't you ever finish anything, don't you want the [crocheted thing] you set out to create?" - and the honest answer is "no".... but if you say that - they ask "well why did you start making it in the first place?" Most people don't seem to understand enjoying the process - at least not on a gut level... I actually solved this particular dilemma by giving away my crocheted things to my grandma - who likes owning crocheted doilies et al. Works for embroidery projects too. Unfortunately, I still tend to get lack of understanding from other people: "but why don't you ever make something for yourself?" I find it very hard to explain to goal-oriented people why I don't like crochet... I like crocheting. I would definitely consider myself to be more process-oriented than goal-oriented. I like doing stuff... I like crocheting, not the goal of having crocheted something in particular. Especially, I like learning - not the feat of "having learned something". So for me - it's very difficult to go to those "attain your goals" seminars etc - because I don't have set goals. I can't point at something and say I want to have achieved precisely that thing, because for me, the thing itself doesn't matter. It can be frustrating, because I cer
HopeFox110

What really struck me with this parable is that it's so well-written that I felt genuine horror and revulsion at the idea of an AI making heaps of size 8. Because, well... 2!

So, aside from the question of whether an AI would come to moral conclusions such as "heaps of size 8 are okay" or "the way to end human suffering is to end human life", the question I'm taking away from this parable is, are we any more enlightened than the Pebblesorters? Should we, in fact, be sending philosophers or missionaries to the Pebblesorter planet to explain to them that it's wrong to murder someone just because they built a heap of size 15?

2Luke_A_Somers
Maybe they just hadn't finished it yet…
HopeFox00

If I actually trust the lottery officials, that means that I have certain knowledge of the utility probabilities and costs for each of my choices. Thus, I guess I'd choose whichever option generated the most utility, and it wouldn't be a matter of "intuition" any more.

Applying that logic to the initial Mugger problem, if I calculated, and was certain of, there being at least a 1 in 3^^^^3 chance that the mugger was telling the truth, then I'd pay him. In fact, I could mentally reformulate the problem to have the mugger saying "If you don't g... (read more)

8Polymeron
I think this is actually the core of the issue - not certainty of your probability, per se, but rather how it is derived. I think I may have finally solved this! See if you can follow me on this... If Pascal Muggers were completely independent instances of each other - that is, every person attempting a Pascal's Mugging has their own unique story and motivation for initiating it, without it correlating to you or the other muggers, then you have no additional information to go on. You shut up and multiply, and if the utility calculation comes out right, you pay the mugger. Sure, you're almost certainly throwing money away, but the off-chance more than offsets this by definition. Note that the probability calculation itself is complicated and not linear: Claiming higher numbers increases the probability that they are lying. However it's still possible they would come up with a number high enough to override this function. At which point we previously said: "Aha! So this is a losing strategy! The Mugger ought not be able to arbitrarily manipulate me in this manner!" Or: "So what's stopping the mugger from upping the number arbitrarily, or mugging me multiple times?" ...To which I answer, "check the assumptions we started with". Note that the assumption was that the Mugger is not influenced by me, nor by other muggings. The mugger's reasons for making the claim are their own. So "not trying to manipulate me knowing my algorithm" was an explicit assumption here. What if we get rid of the assumption? Why, then now an increasingly higher utility claim (or recurring muggings) don't just raise the probability that the mugger is wrong/lying for their own inscrutable reasons. It additionally raises the probability that they are lying to manipulate me, knowing (or guessing) my algorithm. Basically, I add in the question "why did the mugger choose the number 3^^^3 and not 1967? This makes it more likely that they are trying to overwhelm my algorithm, (mistakenly) thinking t
HopeFox10

I can see that I'm coming late to this discussion, but I wanted both to admire it and to share a very interesting point that it made clear for me (which might already be in a later post, I'm still going through the Metaethics sequence).

This is excellent. It confirms, and puts into much better words, an intuitive response I keep having to people who say things like, "You're just donating to charity because it makes you feel good." My response, which I could never really vocalise, has been, "Well, of course it does! If I couldn't make it feel ... (read more)

0BethMo
I'm still going through the Sequences too. I've seen plenty of stuff resembling the top part of your post, but nothing like the bottom part, which I really enjoyed. The best "how to get to paperclips" story I've seen yet! I suspect the problem with the final paragraph is that any AI architecture is unlikely to be decomposable in such a well-defined fashion that would allow drawing those boundary lines between "the main process" and "the paperclip subroutine". Well, besides the whole "genie" problem of defining what is a Friendly goal in the first place, as discussed through many, many posts here.
HopeFox00

I have to say that the sequence on Quantum Mechanics has been awfully helpful so far, especially the stuff on entanglement and decoherence. Bell's Theorem makes a lot more sense now.

Perhaps one helpful way to get around the counterintuitive implications of entanglement would be to say that when one of the experimenters "measures the polarisation of photon A", they're really measuring the polarisation of both A and B? Because A and B are completely entangled, with polarisations that must be opposite no matter what, there's no such thing as "m... (read more)

HopeFox20

It does seem that the probability of someone being able to bring about the deaths of N people should scale as 1/N, or at least 1/f(N) for some monotonically increasing function f. 3^^^^3 may be a more simply specified number than 1697, but it seems "intuitively obvious" (as much as that means anything) that it's easier to kill 1697 people than 3^^^^3. Under this reasoning, the likely deaths caused by not giving the mugger $5 are something like N/f(N), which depends on what f is, but it seems likely that it converges to zero as N increases.

It is a... (read more)