= 27d567bc2d48d9f40e9ccc20ea370b15)
ShardPhoenix comments on Unpacking the Concept of "Blackmail" - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (136)
Isn't the default just what would happen if the other person never communicated with you?
But they did communicate with you, as a result of a somewhat deterministic decision process, and not by random choice. How should you reason about this counterfactual? Why doesn't the "false" assumption of their never communicating with you imply that the Moon is made out of cheese?
People engage in this kind of counterfactual reasoning all the time without declaring the moon to be made of cheese; I'm not sure why you're questioning it here. If it makes it any easier, think of it as being about the change in expected value immediately after the communication vs. the expected value immediately before the communication - in other words, whether the communication is a positive or negative surprise.
Indeed. How do they manage that? That's one fascinating question.
I think they have an underlying premise that they will believe whatever is necessary to make their lives better, or at least not worse.
Their beliefs about what's better and worse may never be examined, so some of their actions may be obviously sub-optimal. However, they won't fall into thinking that one contradiction means they're obligated to believe every piece of obvious nonsense.
When reasoning about counterfactuals a good principle is never to reach to a more distant* world than necessary.
*(less similar)
If you were to simulate the universe as it was before they contacted you, and make 1 single alteration (tapping their brain so they decide not to contact you) would the simulation's moon be made of green cheese?
That universe is pretty much the closest possible universe to ours where they don't contact you.
Why are merely similar worlds ought to be relevant at all? There could be ways of approximate reasoning about complicated definition of the actual world you care about, but actually normatively caring about the worlds that you know not to be actual (i.e. the one you actually care about) is a contradiction of terms.
You asked how to reason about counterfactuals.
I answered.
I'm not sure what you're asking now. Could you please clarify?
The reason I think about counterfactuals is to understand cause and effect. If you change something, then anything to which it is a cause must also change.
But things (such as whether the moon is made of green cheese) which AREN'T caused by the something, would not change
You answered informally. It's easy, and not what I wondered about.
Things don't change. When you make a decision, you are not changing the future, you are deciding the future. The future is what it is given your actual decision, all else is fantasy (logically inconsistent even, because the structure of your own mind implies only one decision, when you ignore some unimportant cognitive noise), perhaps morally important fantasy whose structure we ought to understand, but still not the reality.
What did you wonder about? You seemed to be wondering why you shouldn't just go "Well, if they'd done that, the moon would have been made of cheese".
If you can't think about counterfactuals such as "What will happen if I do X?" "What will happen if I do Y?" etc., you can't make rational decisions.
You may wish to dismiss counterfactuals as fantasy, but does doing so help you come to good decisions? Or does it hinder you?
The goal is not to dismiss counterfactuals, but to understand where they come from, and how are they relevant for reasoning about the actual world. Distinguish inability to reason informally from lack of formal understanding of the structure of that informal reasoning.
They are a mode of thought. They come from the thinker.
They allow you to look at cause and effect. Without counterfactuals, you can't reason about cause and effect, you can only reason about correlation.
Taboo cause, effect, taboo counterfactuals. That something is, doesn't answer why it's normatively useful ("they come from the thinker").
I am not sure that I am correct. But there seems to be another possibility.
If we assume that the world is a model of some formal theory, then counterfactuals are models of different formal theories, whose models have finite isomorphic subsets (reality accessible to the agent before it makes a decision).
Thus counterfactuals aren't inconsistent as they use different formal theories, and they are important because agent cannot decide the one that applies to the world before it makes a decision.
I'm getting this more clearly figured out. In the language of ambient control, we have: You-program, Mailer-program, World-program, Your utility, Mailer utility
"Mailer" here doesn't mean anything. Anyone could be a mailer.
It is simpler with one mailer but this can be extended to a multiple-mailer situation.
We write your utility as a function of your actions and the mailer's actions based on ambient control. This allows us to consider what would happen if you changed one action and left everything else constant. If you would have a lower utility, we define this to be a "sacrificial action".
A "policy" is a strategy in which one plays a sacrificial action in a certain class of situation.
A "workable policy" is a policy where playing it will induce the mailer to model you as an agent that plays that policy for a significant proportion of the times you play together, either for:
causal reasons - they see you play the policy and deduce you will probably continue to play it, or they see you not play it and deduce that you probably won't
acausal reasons - they accurately model you and predict that you will/won't use the policy.
A "beneficial workable policy" is when this modeling will increase your utility.
Depending on the costs/benefits, a beneficial workable policy could be rational or irrational, determined using normal decision theory. The name people use for it is unrelated - people have given in to and stood up against blackmail, they have given in to and stood up against terrorism, they have helped those who helped them or not helped them.
Not responding to blackmail is a specific kind of policy that is frequently, when dealing with humans, workable. It deals with a conceptual category that humans create without fundamental decision-theoretic relevance.
It doesn't (at least not by varying one argument of that function), because of explicit dependence bias (this time I'm certain of it). Your action can acausally control the other agent's action, so if you only resolve uncertainty about the parameter of utility function that corresponds to your action, you are being logically rude by not taking into account possible inferences about the other agent's actions (the same way as CDT is logically rude in only considering the inferences that align with definition of physical causality). Form this, "sacrificial action" is not well-defined.
I think you're mostly right. This suggests that a better policy than 'don't respond to blackmail' is 'don't respond to blackmail if and only if you believe the blackmailer to be someone who is capable of accurately modelling you'.
Unfortunately this only works if you have perfect knowledge of blackmailers and cannot be fooled by one who pretends to be less intelligent than they actually are.
This also suggests a possible meta-strategy for blackmailers, namely "don't allow considerations of whether someone will pay to affect your decision of whether to blackmail them", since if blackmailers were known to do this then "don't pay blackmailers" would no longer work.
I would also suggest that while blackmail works with some agents and not others, it isn't human-specific. For example, poison arrow frogs seem like a good example of evolution using a similar strategy, having an adaptation that is in no way directly beneficial (and presumably is at least a little costly) that exists purely to minimize the utility of animals which do not do what it wants.
Not perfect knowledge, just some knowledge together with awareness that you can't reason from it in certain otherwise applicable heuristic ways because of the incentives to deceive.
Yes, that's what I meant. I have a bad habit of saying 'perfect knowledge' where I mean 'enough knowledge'.
Can I take it that since you criticized a criticism of this hypothesis without offering a criticism of your own, that you believe that this hypothesis is correct?
What hypothesis?
My comment was entirely local, targeting a popular argument that demands perfect knowledge where any knowledge would suffice, similarly to the rhetoric device of demanding absolute certainty where you were already presented with plenty of evidence.
It's evidence that you have seen the comment that he's replying to, in which I lay out my hypothesis for the answer to your original question. (You've provided an answer which seems incomplete.)