EDIT: This was wrong.
The answer varies with the generating algorithm of the statement the guard makes.
In this example, he told you that you were not in one of the places you're not in (the Vulcan Desert). If he always does this, then the probability is 1/4; if you had been in the Vulcan Desert, he would have told you that you were not in one of the other three.
If he always tells you whether or not you're in the Vulcan Desert, then once you hear him say you're not your probability of being in the Vulcan Mountain is 1/3.
Definitely makes sense. A commonly cited example is women in an office workplace; what would be average assertiveness for a male is considered "bitchy", but they still suffer roughly the same "weak" penalties for non-assertiveness.
With the advice-giving aspect, some situations are likely coming from people not knowing what levers they're actually pulling. Adam tells David to move his "assertiveness" lever, but there's no affordance gap available for David by moving that lever -- he would actually have to move an &...
I suspect there's a practise effect here as well. Figuring out how to be assertive without being domineering or bossy is hard. People who have grown up being assertive will have had the opportunity to learn, but those who try to become assertive because they know its important for the workplace won't have developed the judgement yet.
I perceive several different ways something like this happens to me:
1. If I do something that strains my working memory, I'll have an experience of having a "cache miss". I'll reach for something, and it won't be there; I'll then attempt to pull it into memory again, but usually this is while trying to "juggle too many balls", and something else will often slip out. This feels like it requires effort/energy to keep going, and I have a desire to stop and relax and let my brain "fuzz over". Eventually I&#x...
A note on this, which I definitely don't mean to apply to the specific situations you discuss (since I don't know enough about them):
If you give people stronger incentives to lie to you, more people will lie to you. If you give people strong enough incentives, even people who value truth highly will start lying to you. Sometimes they will do this by lying to themselves first, because that's what is necessary for them to successfully navigate the incentive gradient. This can be changed by their self-awareness and force of will, but some wh...
I think I may agree with the status version of the anti-hypocrisy flinch. It's the epistemic version I was really wanting to argue against.
Ok yeah, I think my concern was mostly with the status version-- or rather that there's a general sensor that might combine those things, and the parts of that related to status and social management are really important, so you shouldn't just turn the sensor off and run things manually.
... That doesn't seem like treating it as being about epistemics to me. Why is it epistemically relevant? I thin...
I will see if I can catch a fresh one in the wild and share it. I recognize your last paragraph as something I've experienced before, though, and I endorse the attempt to not let that grow into righteous indignation and annoyance without justification -- with that as the archetype, I think that's indeed a thing to try to improve.
Most examples that come to mind for me have to do with the person projecting identity, knowledge, or an aura of competence that I don't think is accurate. For instance holding someone else to a social standard tha...
As you say, there are certainly negative things that hypocrisy can be a signal of, but you recommend that we should just consider those things independently. I think trying to do this sounds really really hard. If we were perfect reasoners this wouldn't be a problem; the anti-hypocrisy norm should indeed just be the sum of those hidden signals. However, we're not; if you practice shutting down your automatic anti-hypocrisy norm, and replace it with a self-constructed non-automatic consideration of alternatives, then I think you'll do wors...
One hypothesis is that consciousness evolved for the purpose of deception -- Robin Hanson's "The Elephant in the Brain" is a decent read on this, although it does not address the Hard Problem of Consciousness.
If that's the case, we might circumvent its usefulness by having the right goals, or strong enough detection and norm-punishing behaviors. If we build factories that are closely monitored where faulty machines are destroyed or repaired, and our goal is output instead of survival of individual machines, then the machines being dec...
Some reader might be thinking, "This is all nice and dandy, Quaerendo, but I cannot relate to the examples above... my cognition isn't distorted to that extent." Well, let me refer you to UTexas CMHC:
Maybe you are being realistic. Just for the sake of argument, what if you're only 90% realistic and 10% unrealistic? That mean's you're worrying 10% "more" than you really have to.
Not intending to be overly negative, but this is not a good argument for anything and also doesn't answer the hypothetical question of no...
For most questions you can't really compute the answer. You need to use some combination of intuition and explicit reasoning. However, this combination is indeed more trustworthy than intuition alone, since it allows treating at least some aspects of the question with precision.
I don't think this is true; intuition + explicit reasoning may have more of a certain kind of inside view trust (if you model intuition as not having gears that can be trustable), but intuition alone can definitely develop more outside-view/reputational trust. Sometimes...
If you choose to "care more" about something, and as a result other things get less of your energy, you are socially less liable for the outcome than if you intentionally choose to "care less" about a thing directly. For instance, "I've been really busy" is a common and somewhat socially acceptable excuse for not spending time with someone; "I chose to care less about you" is not. So even if your one and only goal was to spend less time on X, it may be more socially acceptable to do that by adding Y as cover.
Social excusability is often reused as internal excusability.
Some reasons this is bad:
I also dislike this comment because I think it's too glib.
I think it's a memetic adaptation type thing. I would claim that attempting to open up the group usage of NVC will also (in a large enough group) open up the usage of "language-that-appears-NVCish-even-if-against-the-stated-philosophy". I think that this type of language provides cover for power plays (re: the broken link to the fish selling scenario), and that using the language in a way that maintains boundaries requires the group to adapt and be skillful enough at detecting these violations. It is not enough if you do so as an individu...
This is incorrect and I think only sounds like an argument because of the language you're choosing; there's nothing incoherent about 1. preferring evolutionary pressures that look like Moloch to exist so that you end up existing rather than not existing, and 2. wanting to solve Moloch-like problems now that you exist.
Also, there's nothing incoherent about wanting to solve Moloch-like problems now that you exist regardless of Moloch-like things causing you to come into existence. Our values are not evolution's values, if that even makes sense.
Upfront note: I've enjoyed the circling I've done.
One reason to be cautious of circling: dropping group punishment norms for certain types of manipulation is extremely harmful. From my experience of circling (which is limited to a CFAR workshop), it provides plausible cover for very powerful status grabs under the aegis of "(just) expressing feelings and experiences"; I think the strongest usual defense against this is actually group disapproval. If someone is able to express such a status grab without receiving overt disapproval, the...
"Complaining about your trade partners" at the level of making trade decisions is clearly absurd (a type error). "Complaining about your trade partners" at the level of calling them out, suggesting in an annoyed voice they behave differently, looking miffed, and otherwise attempting to impose costs on them (as object level actions inside of an ongoing trade/interaction which you both are agreeing to) is not. These are sometimes the mechanism via which things of value are traded or negotiations are made, and may be preferred by both parties to ceasing the interaction.
A potential explanation I think is implicit in Ziz's writing: the software for doing coordination within ourselves and externally is reused. External pressures can shape your software to be of a certain form; for instance, culture can write itself into people so that they find some ideas/patterns basically unthinkable.
So, one possibility is that fusion is indeed superior for self-coordination, but requires a software change that is difficult to make and can have significant costs to your ability to engage in treaties externally. Increased Mana allows you to offset some of the costs, but not all; some interactions are just pretty direct attempts to check that you've installed the appropriate mental malware.
Assuming the money transfer actually takes place, this sounds like a description of gains from trade; the "no pareto improvement" phrasing is that when actually making the trade, you lose the option of making the trade -- which is of greater than or equal value than the trade itself if the offer never expires. One avenue to get actual Pareto improvements is then to create or extend opportunities for trade.
If the money transfer doesn't actually take place: I agree that Kaldor-Hicks improvements and Pareto improvements shouldn't be conflated. It takes social technology to turn one into the other.
I was definitely very confused when writing the part you quoted. I think the underlying thought was that the processes of writing humans and of writing AlphaZero are very non-random; i.e., even if there's a random number generated in some sense somewhere as part of the process, there's other things going on that are highly constraining the search space -- and those processes are making use of "instrumental convergence" (stored resources, intelligence, putting the hard drives in safe locations.) Then I can understand your claim as &quo...
I think you have a good point, in that the VNM utility theorem is often overused/abused: I don't think it's clear how to frame a potentially self modifying agent in reality as a preference ordering on lotteries, and even if you could in theory do so it might require such a granular set of outcomes as to make the resulting utility function not super interesting. (I'd very much appreciate arguments for taking VNM more seriously in this context; I've been pretty frustrated about this.)
That said, I think instrumental convergence is indeed...
On equal and opposite advice: many more people want you to surrender to them than it is good for you to surrender to, and the world is full of people who will demand your apology (and make it seem socially mandatory) for things you do not or should not regret. Tread carefully with practicing surrender around people who will take advantage of it. Sometimes the apparent social need to apologize is due to a value/culture mismatch with your social group, and practicing minimal or non-internalized apologies is actually a good survival mechanic.
If you are high...
Yeah; it's not open/shut. I guess I'd say in the current phrasing, the "but Aumann’s Agreement Theorem shows that if two people disagree, at least one is doing something wrong." is suggesting implications but not actually saying anything interesting -- at least one of them is doing something wrong by this standard whether or not they agree. I think adding some more context to make people less suspicious they're getting Eulered (http://slatestarcodex.com/2014/08/10/getting-eulered/) would be good.
I think this flaw is basically in ...
This is completely awesome, thanks for doing this. This is something I can imagine actually sending to semi-interested friends.
Direct messaging seems to be wonky at the moment, so I'll put a suggested correction here: for 2.4, Aumann's Agreement Theorem does not show that if two people disagree, at least one of them is doing something wrong. From wikipedia: " if two people are genuine Bayesian rationalists with common priors, and if they each have common knowledge of their individual posterior probabilities, then their posteriors must be e...
I found this extremely helpful; it motivated me to go read your entire blog history. I hope you write more; I think the "dark side" is a concept I had only the rough edges of, but one that I unknowingly desired to understand better (and had seen the hints of in other's writing around the community.) I feel like the similarly named "dark arts" may have been an occluding red herring.
The more you shine the light of legibility, required defensibility and justification, public scrutiny of beliefs, social reality that people's ju...
Does this cause any updating in decreasing the likelihood of nightmare scenarios like the one you described?
Effectively no. I understand that you're aware of these risks and are able to list mitigating arguments, but the weight of those arguments does not resolve my worries. The things you've just said aren't different in gestalt from what I've read from you.
To be potentially more helpful, here's a few ways the arguments you just made fall flat for me:
...I only incidentally mention rationality, such as when I speak of Rationality Dojo as a noun. I als
use every deviation from perfection as ammunition against even fully correct forms of good ideas.
As a professional educator and communicator, I have a deep visceral experience with how "fully correct forms of good ideas" are inherently incompatible with bridging the inferential distance of how far the ordinary Lifehack reader is from the kind of thinking space on Less Wrong. Believe me, I have tried to explain more complex ideas from rationality to students many times. Moreover, I have tried to get more complex articles into Lifehack and elsew...
I have not a clue whether this sort of marketing is a good idea. Let me be clear what I mean: I think there's maybe a 30-40% chance that Gleb is having a net positive impact through these outreach efforts. I also think there's maybe a 10-20% chance that he's having a horrific long-term negative impact through these outreach efforts. Thus the whole thing makes me uncomfortable.
So here's some of the concerns I see; I've gone to some effort to be fair to Gleb, and not assume anything about his thoughts or motivations:
I really appreciate you sharing your concerns. It helps me and other involved in the project learn more about what to avoid going forward and optimize our methods. Thank you for laying them out so clearly! I think this comment will be something that I will come back to in the future as I and others create content.
I want to see if I can address some of the concerns you expressed.
In my writing for venues like Lifehack, I do not speak of rationality explicitly as something we are promoting. As in this post, I talk about growing mentally stronger or being in...
I disagree with your conclusion. Specifically, I disagree that
This is, literally, infinitely more parsimonious than the many worlds theory
You're reasoning isn't tight enough to have confidence answering questions like these. Specifically,
In order to actually say anything like the second that's consistent with observations, I expect your physical laws become much less simple (re: Bell's theorem implying non-locality...
Your question of "after finishing the supertask, what is the probability that 0 stays in place" doesn't yet parse as a question in ZFC, because you haven't specified what is meant by "after finishing the supertask". You need to formalize this notion before we can say anything about it.
If you're saying that there is no formalization you know of that makes sense in ZFC, then that's fine, but that's not necessarily a strike against ZFC unless you have a competitive alternative you're offering. The problem could just be that it's an ill-d...
The model is that persistent reflexes interact with the environment to give black swans; singular events with extremely high legal consequence. To effectively avoid all of them preemptively requires training the stable reflexes, but it could be that "editing out" only a few 10 minute periods retroactively would still be enough (those few periods when reflexes and environment interact extremely negatively.) So I think the "very regular basis" claim isn't substantiated.
That said, we cant actually retroactively edit anyways.
Being to vague to be wrong is bad. Especially when you want to speak in favor of science.
I agree, it's good to pump against entropy with things that could be "Go Science!" cheers. I think the author's topic is not too vague to discuss, but his argument isn't strong or specific enough that you should leap to action based solely on it. I think it's a fine thing to post to Discussion though; maybe this indicate we have ideal different standards for Discussion posts?
...There no reason to say "well maybe the author meant to say X" when h
Actually, this illustrates scientific thinking; the doctor forms a hypothesis based on observation and then experimentally tests that hypothesis.
Most interactions in the world are of the form "I have an idea of what will happen, so I do X, and later I get some evidence about how correct I was". So, taking that as a binary categorization of scientific thinking is not so interesting, though I endorse promoting reflection on the fact that this is what is happening.
I think the author intends to point out some of the degrees of scientiificism by w...
I think it would be good to separate the analysis into FGCA's which are always fallacious, versus those that are only warning signs/rude. For instance, the fallacy of grey is indeed a fallacy, so using it as a counter-argument is a wrong move regardless of its generality.
However, it may in fact be that your opponent is a very clever arguer or that the evidence they present you has been highly filtered. Conversationally, using these as a counter-argument is considered rude (and rightly so), and the temptation to use them is often a good internal warning s...
I very much favor bottom-up modelling based on real evidence rather than mathematical models that come out looking neat by imposing our preconceptions on the problem a priori.
(edit: I think I might understand after-all; it sounds like you're claiming AIXI-like things are unlikely to be useful since they're based mostly on preconceptions that are likely false?)
I don't think I understand what you mean here. Everyone favors modeling based on real evidence as opposed to fake evidence, and everyone favors avoiding the import of false preconceptions. It sou...
Formally, you don't. Informally, you might try approximate definitions and see how they fail to capture elements of reality, or you might try and find analogies to other situations that have been modeled well and try to capture similar structure. Mathematicians et al usually don't start new fields of inquiry from a set of definitions, they start from an intuition grounded in reality and previously discovered mathematics and iterate until the field takes shape. Although I'm not a physicist, the possibly incorrect story I've heard is that Feynman path integrals are a great example of this.
Oh yes, it sounds like I did misunderstand you. I thought you were saying you didn't understand how such a thing could happen in principle, not that you were skeptical of the currently popular models. The classes U and F above, should something like that ever come to pass, need not be AIXI-like (nor need they involve utility functions).
I think I'm hearing that you're very skeptical about the validity of current toy mathematical models. I think it's common for people to motte and bailey between the mathematics and the phenomena they're hoping to model, a...
A mathematical model of what this might look like: you might have a candidate class of formal models U that you think of as "all GAI" such that you know of no "reasonably computable"(which you might hope to define) member of the class (corresponding to an implementable GAI). Maybe you can find a subclass F in U that you think models Friendly AI. You can reason about these classes without knowing any examples of reasonably computable members of either. Perhaps you could even give an algorithm for taking an arbitrary example in U and ...
I've enlarged my social circles, or the set of circles I can comfortably move in, and didn't end up on that model. I think I originally felt that way a lot, and I worked on the "feeling like doing a dramatic facepalm" by reflecting on it in light of my values. When dramatic face palms aren't going to accomplish anything good, I examine why I have that feeling and usually I find out it's because my political brain is engaged even when this isn't a situation where I'll get good outcomes from political-style conversation. You can potentially chan...
Agreed on the 2nd paragraph.
Optimally, you'd be have an understanding of the options available, how you work internally, and how other people respond so you could choose the appropriate level of anger, etc. Thus it's better to explore suggestions and see how they work than to naively apply them in all situations.
Seconded! Another phrase (whose delivery might be hard to convey in text) is "Look, I dunno, but anyways..."
Maybe the big idea is to come across as not expressing much interest in the claim, instead of opposing the claim? I think most people are happy to move on with the conversation when they get a "move on" signal, and we exchange these signals all the time.
I also like that this is an honest way to think about: I really am not interested in what I expect will happen with that conversation (even if I am interested in the details of countering your claim.)
I don't know what you mean, but I think I see a lot of people "being polite" but failing at one of these when it would be really useful for them.
For example, you can be polite while internally becoming more suspicious and angry at the other person (#3 and #4) which starts coming out in body language and the direction of conversation. Eventually you politely end the conversation in a bad mood and thinking the other person is a jerk, when you could've accomplished a lot more with a different internal response.
I think your critique of this being only for disagreements that don't matter is too strong, and your examples miss the context of the article.
This is not a suggested resolution procedure for all humans in all states of disagreement; this is a set of techniques for when you already have and want to maintain some level of cooperative relationship with a person, but find yourself in a disagreement over something. Suggestion 5 above is specifically about disengaging from disagreements that "don't matter", and the rest are potentially useful even if it's a disagreement over something important.
P(vulcan mountain | you're not in vulcan desert) = 1/3
P(vulcan mountain | guard says "you're not in vulcan desert") = P(guard says "you're not in vulcan desert" | vulcan mountain) * P(vulcan mountain) / P(guard says "you're not in vulcan desert") = ((1/3) * (1/4)) / ((3/4) * (1/3)) = 1/3
Woops, you're right; nevermind! There are algorithms that do give different results, such as justinpombrio mentions above.