Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

On accepting an argument if you have limited computational power.

22 11 January 2012 05:07PM

It would seem rational to accept any argument that is not fallacious; but this leads to consideration of problems such as Pascal's mugging and other exploits.

I've had a realization of a subconscious triviality: for me to accept an argument as true, it is not enough that I find no error in it. The argument must also be so structured that I would expect to have found an error if it was invalid (or I myself must make such structured version first). That's how mathematical proofs work - they are so structured that finding an error requires little computational power (only knowledge of rules and reliability); in the extreme case an entirely unintelligent machine can check a proof.

In light of this I propose that those who want to make a persuasive argument should try to structure the argument so it'd be easy to find flaws in it. This also goes for the thought experiments and hypothetical situations. Those seem rather often to be constructed with entirely opposite goal in mind - to obstruct the verification process or to try to prevent the reader from trying to find flaws.

Something else tangentially related to the arguments. The faulty models are the prime cause of decision errors; yet the faulty models are the staple of thought experiment; nobody raises an eyebrow as all models are ultimately imperfect.

However, to accept an argument based on imperfect model one must be capable of correctly propagating the error and estimating the error in the final conclusion, as a faulty model may be so constructed as to itself differ non substantially from the reality but in such a way that the difference diverges massively along the chain of reasoning. My example of this is the Trolley Problems. The faults of original model are nothing out of ordinary; simplified assumptions of the real world, perfect information, etc. Normally you can have those faults in model and still arrive at reasonably close outcome. The end result is throwing of fat people onto tracks, cutting up of travellers for organs, and similar behaviours which we intuitively know we could live a fair lot better without. How that happens? In real world the strongly asymmetrical relations of form 'death of 1 person saves 10 people' are very rare (as an emergent property of complexity of the real world that is lacking in the imaginary worlds of trolley problems), while the decision errors are not nearly so rare, so most of people killed to save others would end up killed in vain.

I don't know how models can be structured as to facilitate propagation of model's error. But it seems to be necessary for arguments based on models to be convincing.

Comments (85)

Sort By: Best
Comment author: 11 January 2012 06:48:49PM 20 points [-]

There's a similar guideline in the software world:

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.

Comment author: 11 January 2012 05:30:31PM 7 points [-]

Your first point seems interesting. How specifically should we go about structuring arguments to make flaws easy to find?

Comment author: 11 January 2012 07:37:15PM 6 points [-]

Some ideas:

The logical structure of an argument could be more clearly demonstrated by argument diagramming techniques.

Wherever possible, seek to replace intuition with explicit evidence.

Similarly, try to track the definitions of key concepts, ensuring that every term is fleshed out in more concrete terms, somewhere. Thus, issues of usage are replaced by pointers to specifics.

None of these are new ideas on LW, but they should ask help this goal.

Comment author: 12 January 2012 09:07:48PM 3 points [-]

Well, one important thing to avoid is the type of argument that requires massive exhaustive search through a vast space to find a flaw. It can be perhaps escaped by declaring such arguments void unless the full exhaustive search has been performed by a computer.

I had some thoughts for the models. An argument based on the imperfect model of real world should outline imperfections of the model and then propagate the imperfections along with the chain of reasoning, as to provide upper bound on the final error. It is often done with computer simulations of e.g. atmosphere, where you can e.g. run simulation a lot of times with different values that are within the error range and look at the spread of the outcome.

Comment author: 11 January 2012 06:00:47PM 2 points [-]

Use precise language, with explicit and/or clear definitions?

I admit that "Write better" is not often helpful advice, but it is good advice.

Comment author: 11 January 2012 05:30:02PM *  11 points [-]

To me the solution to this problem is to not rely too much on raw consequentialism for dealing with real-life situation. Because I know my model of the world is in perfect, that I lack computing power to track all the consequences of an action and evaluate their utility, because I don't even know my own utility function precisely.

So I'm trying to devise ethical rules that come partly from consequentialism, but also taking into consideration lessons learned from history, both my own personal experience and humanity's history. And those rules for example say I should not kill someone, even if I think it'll save 10 lives, because usually when you do that, either you kill the person and fail to save the 10 others, or you failed to think to a way to save the 10 without killing one, or you create far-reaching consequences that'll at the end cost more than the 10 saved lives (for example, breaking the "don't kill" taboo, and leading for people to follow your example even in cases when they'll fail to save the 10 persons). That's less optimal than using consequentialism wisely - but also much less error-prone, at least to me, than trying to wield a dangerous tool that I'm not smart/competent enough to wield.

That's quite similar to the way we can't use QM and GR to make planes, but we use simpler, "higher-level" laws, which are not as precise, but much more computable, and good enough for our needs. I acknowledge the core of physics is QM and GR, and the rest are just approximations, but we use the approximations because we can't wield raw QM and GR for most daily life problems. And I acknowledge consequentialism to be the core of ethics, but I do think we need approximations, because neither can we wield directly consequentialism in real life.

Comment author: 12 January 2012 05:47:57AM 1 point [-]

More precisely, the core of our current best available (but still known to be flawed) physics are QM and GR and we do not even have a consistent model fully incorporating both.

Comment author: 17 January 2012 05:17:40AM 2 points [-]

Furthermore, we can't model anything more complicated then a hydrogen atom with QM without resorting to approximations, and by the time you get to something as complicated as bulk matter or atomic nuclei of heavy elements, we can't even verify that the predictions of QM are what we in fact observe.

Comment author: 17 January 2012 05:25:40AM 0 points [-]

Very true, but we can test at least some multiple-particle predictions by attempting to build a small quantum computer

Comment author: 13 January 2012 09:23:18PM 0 points [-]

From what I understand, we have more than one. We just don't know which, if any, is correct.

Comment author: 17 January 2012 04:37:25AM 0 points [-]

We have some plans (including a few radically different from everything we are used to - which is good) how to build a model. I wouldn't call these plans models of anything yet, because QM and GR can help us predict the behaviour of precise tools we use, and these plans are not yet concrete enough to allow useful modelling.

Comment author: [deleted] 14 January 2012 12:01:02AM 0 points [-]

And some of them have so damn many free parameters that it would be hard to rule them out but they have hardly any predictive power.

Comment author: 13 January 2012 09:29:15PM -1 points [-]

I think the biggest problem with killing someone is that you're likely to get arrested, which prevents you from saving hundreds of lives.

In general, the best way to get something done is to pay someone who's better at it than you. As such, you can fairly accurately simplify it into thinking about how to earn money, and thinking about where to donate that money. These are generally things you can think about once, rather than think about on a case-by-case basis.

Comment author: 14 January 2012 04:15:34PM 1 point [-]

That specialization of labor does a lot of help doesn't mean that extreme specialization still does a lot of help. There are so many issues involved with letting someone else do something for you (finding/chosing the person, trust, explaining what you have to do, moving the person to the place, schedule/calendar issues, negotiating the price, legal issues, ...) that for many things, it's less efficient to pay someone to do it than to do it yourself, even if for the core of the task, a specialized person would be more efficient.

Also, you've to consider willpower/akrasia/enjoyment related issues. For example, many people will feel much more motivated when fixing your own house than when fixing someone else house, so even if you need more time to do it than a professional, you could still fill better doing it yourself, than working (even less) extra hours in your job and paying someone to do it. Oversimplifying things like that just doesn't work in real life.

And finally, you've to consider emergency situations. Trolley like situations are emergency situations, like if you see someone being mugged, or someone drowning of whatever, in those, you just don't have the option to pay someone to act for you.

Comment author: 14 January 2012 05:49:53PM 0 points [-]

There are so many issues involved with letting someone else do something for you (finding/chosing the person, trust, explaining what you have to do, moving the person to the place, schedule/calendar issues, negotiating the price, legal issues, ...) that for many things, it's less efficient to pay someone to do it than to do it yourself

There would be vastly more things like this if specialization wasn't normal. That's what I meant when I said that it works better the more it's done. There are things it's better to do yourself, but most things aren't like that.

And finally, you've to consider emergency situations.

The benefits from a situation not covered by the rule I gave earlier are very small. If you're in a situation where acting would be remotely dangerous, don't. If acting would be perfectly safe, go ahead. If it would be very slightly dangerous, then you're likely to be better off doing a Fermi calculation.

Comment author: 13 January 2012 09:42:11PM 0 points [-]

In general, the best way to get something done is to pay someone who's better at it than you. As such, you can fairly accurately simplify it into thinking about how to earn money

I'm not sure this generalizes well - would this work if everybody was doing it? (It might).

Comment author: 14 January 2012 02:06:23AM 0 points [-]

Without specialization of labor, the world simply would not support this many people. Billions would die.

It generalizes well. In fact, the more people do it, the better it works.

Comment author: 14 January 2012 05:30:43PM 0 points [-]

But I wonder whether thinking "How can I earn money?" gets specialization of labor as well / better than thinking "What's interesting to me?"

It might result in people trying and failing to do things that pay a lot, rather than try and succeed at things they're well-suited for.

Comment author: 14 January 2012 05:38:59PM 0 points [-]

If you know that will be a problem, I think you're smart enough to figure out that you have to do something you're interested in. If not, you're not going to come up with this as a guideline in the first place.

Comment author: 12 January 2012 12:24:29AM 2 points [-]

The issues brought up here impinge upon a concern of my own regarding issues that many Less Wrongers accept.

I don't know if I can let myself invest belief in things such as the Singularity (after I figure out what I mean) or cryonics without working some calculations out for myself.

The problem is that I don't know how to do that; and moreover, if I did, I'm worried that trying to do that would have so many possible points of error (and things to overlook) as to invariably give me overconfidence in whatever I was already leaning towards, with an anchoring bias towards acceptance given the posts on LW. Does anyone have any thoughts on this?

Comment author: 12 January 2012 12:57:52AM 1 point [-]

To me singularity and cryonics are two different beasts.

The singularity is too far into the fog of the future to be taken seriously today. Maybe when there are some significant advances in the relevant fields, whenever that might happen. So far, even a single neuron is not fully simulated. The progress might go so many different ways, I find the singularity being just one of the many possible directions, and I know that my imagination is not up to par to even consider most of them.

On the other hand, cryonics is basically a bet that a frozen brain can potentially be fully restored. We already know that single cells can be (sperm banks do it on an industrial scale), and there is some success with organs and even organisms. So we might be only a few steps away from being able to repair the damage done by freezing.

Comment author: 12 January 2012 05:02:27AM 2 points [-]

I think this may have been given a down vote because to some a Singularity seems settled and rational. I take it to be the idea that eventually--inevitably--machines will become smart enough to improve themselves, that once they become just a bit smarter than human programmers and with the capacity of self-modification, that it will be a runaway phenomenon.

And that's why FAI is such a big deal: precisely because it's "far into the fog of future" in the sense that we don't have any control on the outcome (unless we align its values very precisely with our own).

But here's the rub - for me, anyway - might it be rational to take your own reasoning as suspect about such 'foggy' matters? That is, even if you are doing your best to reason rationally, might it be best to assume that your potential knowledge is reduced to very little and the remainder filled in by cognitive bias?

Okay, maybe this is a better approach---since questions like Singularity&FAI are potentially so important---can anyone point me to a resource that convinced them of its reality?

Comment author: 12 January 2012 06:01:37AM *  2 points [-]

I would add, a resource that does not rely on the teachings of the local prophet, since most of us have already read those.

Comment author: 11 January 2012 05:37:49PM *  2 points [-]

similar behaviours which we intuitively know we could live a fair lot better without

The situations presented are indeed such ones that we could live better without, but the whole point of thought experiments is to construct the worst possible world, and find a way to decide that works even under those circumstances. By your logic, we could easily end up saying "it's useless to argue about how one or two electrons behave, real world objects have much more of them... and anyway, tunneling effects are so weird and unintuitive that we surely have a wrong model".

So while I agree that faulty models often result in faulty decisions, the question to answer is whether the model correctly applies to the current situation, and not whether the results are intuitive or not. Sometimes they really aren't...

(edit: that said, you're right in that before accepting such an unintuitive consequence, it's worth thinking it thorough another time. We're humans, after all, running on flawed hardware and having limited time...)

Comment author: 11 January 2012 09:59:09PM *  1 point [-]

Well, the thought experiments are precisely the experiments in what's intuitive; that's why they are thought experiments rather than real experiments. It indeed is fairly useless to argue how one or two electrons behave, you have to take a look in the real world and see.

Likewise with the trolley problems; the implicit presumption is that they are not completely irrelevant to the real world, yet they are because they neglect the false positive rate (assuming it to be 0) while discussing an extremely low probability event (whose probability is well below any plausible false positive rate). This sort of thing is precisely why Bayesian reasoning is so important.

Comment author: 11 January 2012 07:24:04PM *  3 points [-]

The faulty models are the prime cause of decision errors; yet the faulty models are the staple of thought experiment...

The purpose of thought experiment is to analyse our theories in extreme situations. The understanding this gives can then be useful in non-extreme situations. An analogy to mathematics: When graphing y=x/(x^2+1) it is useful to consider the value of y as x goes to infinity, even if we only need a sketch for -2<x<2. Trolley Problems allow us to focus on the conflict between intuition and utilitarianism. The understanding thought experiments bring is important, even if their circumstances do not occur naturally. Indeed, the very fact that their circumstances do not occur naturally is what necessitates a thought experiment.

Comment author: 11 January 2012 09:46:16PM *  2 points [-]

Well, we don't blame special relativity for seeming to fail with all the thought experiments involving objects moving faster than speed of light. edit: that is to say, it is important that thought experiments remain within certain bounds. In the case of the trolley problems, the small difference between the assumptions and real world (neglecting the small false positive rate while focussing on the extremely low probability scenario) turn out to lead to massively incorrect result which is then counter intuitive. edit: and indeed, there are things which are correct but counter intuitive. However most of the things which are counter intuitive are also wrong; Earth being a torus is very counter intuitive and wrong, ditto for the saddle shape, etc.

Comment author: 12 January 2012 08:33:08PM 0 points [-]

That's hardly a critique of the trolley problem. Special relativity itself stipulates that it doesn't apply to faster-than-light movement, but a moral theory can't say "certain unlikely or confusing situations don't count". The whole point of a moral theory is to answer those cases where intuition is insufficient, the extremes you talk about. Imagine where we'd be if people just accepted Newtonian physics, saying "It works in all practical cases, so ignore the extremes at very small sizes and very high speeds, they are faulty models". Of course we don't allow that in the sciences, so why should we in ethics?

Comment author: 12 January 2012 09:01:06PM *  2 points [-]

In the practical reasoning, "A" is a shorthand for "I think A is true", et cetera - no absolute knowledge, nonzero false positive rate, and sufficiently refined moral theory has to take this into account.

Just as thought experiment relying on e.g. absolute simultaneity would render itself irrelevant to special or general relativity, so does trolley problem's implicit assumption of absolute, reliable knowledge render it irrelevant to the extreme cases where the probability of event is much smaller than false positive rate.

Comment author: 13 January 2012 12:33:01AM -1 points [-]

The analogy between moral theories and physics seems to suggest that just as we expect modern physics to act like Newtonian physics when dealing with big slow objects, we should expect some modern moral theory to act like folk morality when dealing with ordinary human life situations. Does that hold?

Comment author: 11 January 2012 07:29:52PM 2 points [-]

Yes, though it is a standard response that weird situations are weird, and we should not expect our tools for dealing with everyday situations to apply. (cf. the ethical literature on "desert island cases")

Comment author: 11 January 2012 07:41:03PM 3 points [-]

I'd respond to that thusly: In order to extend our tools a little bit (as is necessary to deal with new situations), it helps to imagine extending them further than necessary. Otherwise multiple extensions in series will render them hopeless.

Comment author: [deleted] 12 January 2012 03:30:39PM 0 points [-]

This is an excellent point, but wouldn't making a model as transparent as possible require an attention to one's audience? If so, wouldn't that preclude any method or rule about how to make a model transparent?

Comment author: 12 January 2012 01:57:38AM 0 points [-]

Is the following a loosely accurate summary: "There are certain types of arguments where the probability of the argument not having any obvious flaws even if it is wrong is high enough that one shouldn't count the arguments as significant evidence for their conclusions."

Comment author: 12 January 2012 01:46:56AM 0 points [-]

Who are you calling an "entirely unintelligent machine"? I think I'm offended.

Comment author: 16 January 2012 10:56:44PM 0 points [-]

Computer running automatic theorem prover ;)

Comment author: 11 January 2012 05:56:05PM 0 points [-]

One approach is to apply the scientific method: every model must have testable predictions. For the case of Pascal's mugging, you can ask the mugger to demonstrate their "magic powers from outside the Matrix" in a benign but convincing way ("Please show me a Turing machine that simulates an amoeba"). If they refuse or are unable to, you move on.

Comment author: 12 January 2012 01:59:44PM *  3 points [-]

The point of Pascal's Mugging dealing with a tiny probability of a really big harm vs. a high probability of a very small harm. If your mugger makes testable predictions about his power to carry out the really big harm, that turns the tiny probability into a reasonably large probability, and makes it a non-Pascalian hostage situation.

Comment author: 12 January 2012 05:36:33PM 0 points [-]

When confronted with highly speculative claims so beloved by philosophers, string theorists and certain AI apologists, my battle cry is "testable predictions!". If one argues in favor of a model that predicts a tiny probability of a really big harm, they better provide a testable justification of that model. In the case of Pascal's mugging, I have suggested a simple way to test if the model should be taken seriously. Such a test would have to be constructed specifically for each individual model, of course. If all you say is "I can't prove anything, but if I'm right, it'll be really bad", I yawn and move on.

Comment author: 12 January 2012 06:17:35PM 2 points [-]

If all you say is "I can't prove anything, but if I'm right, it'll be really bad", I yawn and move on.

This is the normal response, even here at LW--I think there's a popular misperception that LW doctrine is to give the Pascal's Mugger money. The point of the exercise is to examine the thought processes behind that intuitive, obviously correct "no," when it appears, on the surface, to be the lower expected utility option. After all, we don't want to build an AI that can be victimized by Pascalian muggers.

One popular option is the one you picked: Simply ignore probabilities below a certain threshhold, whatever the payoff. Another is to discount by the algorithmic complexity, or by the "measure" of the hostages. Yet another is to observe that, if 3^^^^3 people exist, a random person' (your) chances of being able to affect all the rest in a life-and-death way has to be scaled by 1/3^^^^3. Yet another is that, in a world where things like this happen, a dollar has near-infinite utility. Komponisto suggested that the kolmogorov complexity of 3^^^^3 deaths, or units of disutility, is much higher than that of the number 3^^^^3; so any such problem is inherently broken.

Of course, if you're not planning to build an optimizing agent, your "yawn and move on" response is fine. That's what the problem is about, not signing up for cryonics or donating to SI or whatever (the proponents of the last two argue for relatively large probabilities of extremely large utilities).

Comment author: 12 January 2012 06:39:32PM 1 point [-]

To possibly expand on khafra's point, Pascal's Mugging is basically a computational problem. An expected utility maximizer, given a reward on the order of 3^^^3, would need a probability of less than 1/3^^^3 to discount it. But given limited computational resources and limited knowledge, there isn't an obvious algorithm that both calculates expected utility properly in normal cases, and actually arrives at a probability of 1/3^^^3 in the case of Pascal's Mugging (one might need more information than is in the universe to justify a probability that small).

Comment author: 12 January 2012 07:31:32PM 0 points [-]

In the "least convenient possible world" one would reduce the number of people to something that does not stretch the computational limits. I believe that my argument still holds in that case.

Comment author: 12 January 2012 07:51:17PM 0 points [-]

I'm confused. Is your invocation of LCPW supposed to indicate something that would solve this particular decision theory problem? If so, can you provide an algorithm that will successfully maximize expected utility in general and not fail in the case of problems like Pascal's Mugging?

Comment author: 12 January 2012 09:46:15PM 0 points [-]

Since 3^^^3 is unfeasible, suppose the mugger claims to simulate and kill "only" a quadrillion humans. The number is still large enough to overload one's utility, if you assign any credence to the claim. I am no expert in decision theory, but regardless of the exact claim, if the dude refuses to credibly simulate an amoeba, your decision is simple: ignore and move on. Please feel free to provide an example of Pascal mugging where this approach (extraordinary claims require extraordinary evidence) fails.

Comment author: 12 January 2012 11:00:36PM *  2 points [-]

Pascal's mugging only works if after some point your estimated prior for someone's ability to cause utilitarian losses of size n decreases more slowly than n increases; otherwise, claims of extravagant consequences make the mugging less likely to succeed as they grow more extravagant. "Magic powers from outside the Matrix" fill that role in the canonical presentation, since while the probability of that sort of magic existing is undoubtedly extremely small we don't have any good indirect ways of estimating its probability relative to its utilitarian implications, and we can't calculate it directly for the reasons thomblake gave a few comments up.

A quadrillion humans, however, don't fit the bill. We can arrive at a reasonable estimate for what it'd take to run that kind of simulation, and we can certainly calculate probabilities that small by fairly conventional means: there's a constant factor here that I have no idea how to estimate, but 1 * 10^-15 is only about eight sigma from the mean on a standard normal distribution if I got some back-of-the-envelope math right. I'd feel quite comfortable rejecting a mugging of that form as having too little expected damage to be worth my time.

Comment author: 13 January 2012 02:55:31AM 0 points [-]

I must be missing something. To me a large number that does not require more processing power/complexity than the universe can provide is still large enough. TBH, even 10^15 looks to me too large to care, either the mugger can provide reasonable evidence or not, that's all that matters.

Comment author: 13 January 2012 03:27:45AM *  1 point [-]

If the mugger can provide reasonable evidence of his claims, it's not a decision-theoretically interesting problem; instead it becomes a straightforward, if exotic, threat. If the claim's modest enough that we can compute its probability by standard means, it becomes perfectly normal uncreditable rambling and stops being interesting from the other direction. It's only interesting because of the particular interaction between our means of updating probability values and a threat so fantastically huge that the expected loss attached to it can't be updated into neutral or negative territory by observation.

Comment author: 12 January 2012 10:02:29PM 0 points [-]

You can assign credence to the claim and still assign little enough that a quadrillion humans won't overload it. I think the claim the be able to simulate a quadrillion humans is a lot more probable than the claim to be able to simulate 3^^^3 (you'd need technology that almost certainly doesn't exist, but not outside-the-Matrix powers,) but I'd still rate it as being so improbable as to only account for a tiny fraction of an expected death.

Comment author: 12 January 2012 10:22:11PM 0 points [-]

I'm settling for just one quadrillion to avoid dealing with the contingency of "3^^^3 is impossible because complexity". The requirement of testability is not affected by the contingency.

Comment author: 13 January 2012 02:58:25AM 0 points [-]

If you assign the threat a probability of, say, 10^-20, the mugger is extorting considerably more dead children from you than you should expect to die if you don't comply.

Comment author: 12 January 2012 06:20:34PM 0 points [-]

And in the least convenient worlds?

Comment author: 12 January 2012 07:21:57PM 0 points [-]

So I recommend: limit yourself to responses of the form "I completely reject the entire basis of your argument" or "I accept the basis of your argument, but it doesn't apply to the real world because of contingent fact X." If you just say "Yeah, well, contigent fact X!" and walk away, you've left yourself too much wiggle room.

Which contingent fact X do you mean?

Comment author: 12 January 2012 07:29:26PM 0 points [-]

That you can demand testing in many real world scenarios, a heuristic not always usable.

Or do you have a principled decision theory in mind, where testing is a key modification to the equations of expected-value etc and which defuses the mugging?

Comment author: 12 January 2012 09:52:44PM 0 points [-]

That you can demand testing in many real world scenarios, a heuristic not always usable.

As a natural scientist, I would refuse to accept untestable models. Feel free to point out where this fails in any scenario that matters.

Comment author: 12 January 2012 10:43:34PM 0 points [-]

<insert standard anti-logical positivism argument like 'how do you test unreproducible events like "human history"?'>

Comment author: 12 January 2012 10:00:14PM 0 points [-]

How do you determine if the model is testable? What if there is in principle a test, but it has unacceptable consequences in at least one reasonably probable model?

Comment author: 12 January 2012 10:27:52PM 0 points [-]

For the particular scenario described in the Pascal's mugger, I provided a reasonable way to test it. If the mugger wants to dicker about the ways of testing it, I might decide to listen. It is up to the mugger to provide a satisfactory test. Hand-waving and threats are not tests. You are saying that there are models where testing is unfeasible or too dangerous to try. Name one.

Comment author: 13 January 2012 06:56:09PM 1 point [-]

That such models exist is trivial - take model A, add a single difference B, where exercising the difference is bad. For instance,

Model A: universe is a simulation Model B: universe is a simulation with a bug that will crash the system, destroying the universe, if X, but is otherwise identical to model A.

Models that would deserve to be raised to the level of our attention in the first place, however, will take more thought.

Comment author: 12 January 2012 05:32:50PM 0 points [-]

Conversely, the inability of a putative Pascal's Mugger to make such predictions ought to apply a significant penalty to the plausibility of its claim. And simply increasing the threatened disutility won't necessarily help, since the more powerful the entity claims to be, the greater should be the implausibility of its inability to make testable predictions.

Comment author: 12 January 2012 07:54:03AM 1 point [-]

Well, that sometimes makes it more awkward to design the thought experiment. See the Least Convenient Possible World.

On the one hand, I have to give my friend credit: his answer is biologically accurate, and beyond a doubt the technically correct answer to the question I asked. On the other hand, I don't have to give him very much credit: he completely missed the point and lost a valuable effort to examine the nature of morality.

Comment author: 11 January 2012 10:15:44PM *  1 point [-]

The point would be that model has to be easily falsifiable - you should be able to see if it doesn't simulate amoeba.

The Pascal's mugging is not easily falsifiable. I think of myself as a fairly intelligent individual; it took me 4 days to think that - whoah, the number of possible beings is not as huge as 3^^^^^3 ; if a being is made of 10^30 components each of which can be in either of 10^30 states, the number is only (10^30)^(10^30) which seems way smaller (not sure though, the knuth's up arrow is unwieldy to compare to anything). And that's some seriously frigging complex being I'm talking of (to human as human is to amoeba). I can't rely on myself to falsify such stuff reliably.

Comment author: 12 January 2012 12:38:26AM -1 points [-]

I can't rely on myself to falsify such stuff reliably.

You don't need to, as they would most likely be unable to simulate an amoeba, so you can call their bluff right there.

Comment author: 12 January 2012 08:48:03PM 1 point [-]

The issue with pascal's mugging is that the refusal to simulate amoeba only decreases the probability of validity of the claim by a constant factor, and the mugger can stack more uparrows. Suppose that one in ten thousands superbeings refuses to demonstrate to you the simulated amoeba despite being capable of doing so.

Also, what if it simulated amoeba but claims it'll torture 3^^^^^3 humans? I can falsify the latter part because there can not be this many distinct human mind states, but only after taking a lot of time to think about the claim.

Comment author: 12 January 2012 09:22:26PM 1 point [-]

It seems reasonable to me that my confidence that the mugger can do what it claims to should be inversely proportional to the discrepancy between what the mugger claims and what the mugger is willing to demonstrate. Why do you say it should be a constant factor?

Comment author: 13 January 2012 12:40:18AM *  0 points [-]

What if one in ten thousands genuine super-beings is not a nice guy and refuses to demonstrate you stuff? The refusal is only linked to ability for those unable to demonstrate, or those barely able to demonstrate. If you mail world's top mathematicians a question what is 2*3+4 , very few will write you back; you may actually get higher willingness to 'demonstrate' ability to multiply and add from the elementary school kids.

Comment author: 13 January 2012 01:37:44AM 0 points [-]

True. OTOH, very few of them will approach me with a "mathematician's mugging," either.

Comment author: 13 January 2012 12:28:52AM -1 points [-]

Agreed. Humans already implement this: if someone keeps making bigger and bigger unproven claims, eventually they get a reputation as a braggart and a liar, and their claims don't count for anything.

Comment author: 12 January 2012 10:39:34AM 0 points [-]

When shown that model, how would you judge whether it really simulates an amoeba? It's fairly easy to create a program whose output resembles an amoeba on surface without simulating the movement of all its molecules.

Comment author: [deleted] 12 January 2012 03:12:46PM *  1 point [-]

It occurs to me that I may have made my answer to your post a little long,but I figured that when I was unsure of how to judge something, more tests are better than fewer tests.

Starting with the basic tests from Wikipedia:

1: Does the model have organelles?

2: Does the model have cytoplasm?

3: Are the models organelles and cytoplasm enclosed by a cell membrane?

4: Does the model obtain its food through phagocytosis?

5: Is it a Heterotroph?

6: Does it have a single large tubular pseudopod at the anterior end, and several secondary ones branching to the sides?

It would probably be reasonable for a fake amoeba simulation to have all of these.

Then you start applying more tricky bits, like:

7: "Amoeba's are supposed to be able to recover from a forcible division if the nucleus is intact. If I remove some of the cytoplasm and keep the nucleus intact, does the simulation recover?"

8: "Cytoplasm is supposed to contain water. Does the bit of cytoplasm I extracted contain water?"

9: "Does the water divide into Hydrogen and Oxygen via electrolysis?"

Also, be sure to test things where the answer is expected to be no, as well, in case the simulation is just programmed to spit out "Yes." repeatedly.

10: "Does the Amoeba contain multiple nuclei?"

11: "Does the Amoeba have a human nervous system?"

12: "Is the Amoeba more than 99% Salt?"

Then ask 3 more questions which are LIKE question 7-12, but which I am not writing down, in case the person simply Googled for "Questions people have ever asked about Amoebas." and recorded appropriate answers.

Now make sure to ask questions which aren't Yes/No questions, because Yes/No questions don't actually accumulate bits of evidence all that quickly.

For instance

16: "How many Base pairs does the Ameoba have in it's genome?"

17: "How long is the Amoeba while moving?"

18: "How long is the Amoeba while not moving?"

19: "How fast can the Amoeba move?"

20: "Amoebas contain water. Water can put out forest fires. How many of your simulated Amoebas would I have to drop from 100 meters above a forest fire currently covering 100 acres of burning forest surrounded by 1000 acres of unburnt forest to put out the fire before the unburnt forest burns up?" (This seems like a bizarre question, but I am trying to be thorough. Also, someone's reaction to a question that they would never be expected to have an answer to under normal circumstances can itself be revealing.)

And again, ask a few more unlisted questions here, as well, including at least one other question just as odd as the forest fire question. (again, they might have prepared rote answers to all known Ameoba related items.) Also, you may want to ask some other later questions first if you suspect you might have a limited number of questions.

Now it is possible that the person will say "I'm only simulating an Amoeba, there isn't a simulated forest that I can run forest fire tests on." and "I'm only simulating an Amoeba, there isn't anything in the simulation which can be used to generate electricity to do electrolysis on water." and "I'm only simulating an Amoeba, there isn't anything in the simulation which can be used to forcibly divide it." And you would expect SOME of these. For instance, I would be surprised if anyone who was simulating an Amoeba ALSO simulated a forest for forest fire tests using the water inside that Amoeba. That might for instance indicate that they DID have unlimited computing power and simply simulated everything. Or that they were faking answers. It depends on the amount of time the answer takes and the type.

But you could say that to anything. "I'm only simulating an Amoeba, there isn't any light in the simulation to see it." "I'm only simulating an Amoeba, there are no objects in the simulation to test it's permeability to objects." "I'm only simulating an Amoeba, there is nothing in the simulation which can be used to judge how fast the Amoeba is."

If the mugger seems to be saying that or something similar to EVERYTHING, it's very probably a fake, particularly if it sounds almost like a rote answer with no delays.

Given an answer to all of those questions, and a list of objections if any, and timing knowledge about the questions, I think I would have a fairly good grasp of whether the model simulating an Amoeba was valid (or simulating anything really, I can change the specific questions given something else.) given about 25 questions or so. Of course there are edge cases. It is entirely possible to come up with a list of answers which I will see, and then come to the wrong conclusion about. but it should take enough effort that it would be easier to get the mugged amount of money from someone else rather than making a series of incredibly elaborate ruses just for a few dollars.

Comment author: 12 January 2012 05:22:33PM 0 points [-]

Thank you, you have illustrated my point that the test to even consider the mugger seriously, let alone take his claims at face value is easy to devise, but hard to pass, unless one is indeed an expert in simulating living things very convincingly.

Comment author: 16 January 2012 11:08:39PM *  0 points [-]

But there is nonzero probability that the genuine being outside the matrix would not be able to provide satisfactory demonstration due to e.g. limited bandwidth channel it may have to our universe (suppose that someone else is in charge of our universe).

(If that being is omnipotent in our universe you can just ask it to show the flaming text in front of your eyes, no need to simulate amoebas. But if it can demonstrate this, then it is straightforwardly a God and there's no paradox or question of any kind)

Comment author: 17 January 2012 12:51:32AM *  1 point [-]

show the flaming text in front of your eyes [...] if it can demonstrate this, then it is straightforwardly a God

Or a capable trickster. Which one do you think is more likely?

Comment author: 17 January 2012 09:44:30AM 0 points [-]

Say, I were to make a flaming text appear right in the middle of your visual field, moving with your head as you turn it around, like right as you're reading this sentence.

Most likely explanation would be that you are hallucinating, at which point you can just as well give up and lie still for some hours hoping for image to go away.

Comment author: 12 January 2012 09:26:09PM -1 points [-]

I like the first argument more than the second.

"How do we know that not donating to this one poor child with a rare cancer will actually save a bunch of Africans?" would be justified, even though if you're going for lives saved you really should just go with the Africans. Now, it helps that we have metrics, but you still have to decide to do something ever.