LESSWRONG
LW

Comment Permalink

In support of your point, MIRI itself changed (in the opposite direction) from its former stance on AI research.

You've been around long enough to know this, but for others: The former ambition of MIRI in the early 2000s—back when it was called the SIAI—was to create artificial superintelligence, but that ambition changed to ensuring AI friendliness after considering the "terrible consequences [now] feared by the likes of MIRI".

In the words of Zack_M_Davis 6 years ago:

(Disclaimer: I don't speak for SingInst, nor am I presently affiliated with them.)

But recall that the old name was "Singularity Institute for Artificial Intelligence," chosen before the inherent dangers of AI were understood. The unambiguous for is no longer appropriate, and "Singularity Institute about Artificial Intelligence" might seem awkward.

I seem to remember someone saying back in 2008 that the organization should rebrand as the "Singularity Institute For or Against Artificial Intelligence Depending on Which Seems to Be a Better Idea Upon Due Consideration," but obviously that was only a joke.

I've always thought it's a shame they picked the name MIRI over SIFAAIDWSBBIUDC.

Or maybe because SIAI realized their ability to actually create an AI is non-existent

7

Stupid Questions February 2017

8th Feb 2017

1 min read

7

This thread is for asking any questions that might seem obvious, tangential, silly or what-have-you. Don't be shy, everyone has holes in their knowledge, though the fewer and the smaller we can make them, the better.

Please be respectful of other people's admitting ignorance and don't mock them for it, as they're doing a noble thing.

To any future monthly posters of SQ threads, please remember to add the "stupid_questions" tag.

7

Stupid Questions February 2017

1Douglas_Knight

New Comment

105 comments, sorted by

Click to highlight new comments since: Today at 10:56 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

I am getting to grips with the basics of Bayesian rationality and there is something I would like to clarify. For this comment please assume that whenever I use the word 'rationality' I mean 'Bayesian rationality'.

I feel there is too strong a dependency between rationality and available data. If current understanding is close to the truth then using rational assessment will be effective. But in any complex subject the data is so inconclusive that the possibility that we can not even conceive the right hypothesis, to rationally choose it from its alternativ... (read more)

9moridinamael8y

I try to always include an extra option which is just "everything I haven't thought of yet", and the prior for that option reflects how confident you feel about your mastery of the domain. Even a medical expert must always admit some small likelihood that the true cause is neither a tumor nor a cold, because medical experts know that medical science evolves over time.

1Erfeyah8y

That is good. But how would you assign the probability of 'everything you haven't thought of yet' in a problem? You would have to base it again on data; which takes us back to the same question. Data can of course be sufficient for fields that are advanced to a point of overwhelming clarity. But when we get into sufficiently complex subjects the rational approach would be to say that my 'everything I haven't thought of yet' prior is so large that the only rational answer is "I don't know". Would you agree?

3moridinamael8y

Yes, you can't get away from data. And in fact it is often rational to put a dominant prior on "whatever the explanation is, it probably isn't one of the ones that I've managed to come up with". There's nothing magic about Bayes theorem, it can't create new information, but when properly used it helps avoid overconfidence and underconfidence.

1Erfeyah8y

Yes, this is clear, thank you. I recall rational discussions in LW with expositions on ontology, cosmology, sociology, the arts etc. in which probabilities are thrown around as if we know a lot, which makes me think that this point has not been fully understood. On the other hand whenever I talk to people on a one to one bases (like in this comment thread) people seem lucid about it. Will have to address it where I see it. Thanks for your input! :)

0ChristianKl8y

"I don't know" means "I can't predict the outcome better than a dart throwing monkey". Doctors usually know more than that. You seem to have a lot of black and white terms like "right hypothesis", "accepting explanations as truths" and "I don't know" in your thinking. In the Bayesian perspective 0 and 1 aren't probabilities.

0Erfeyah8y

The example I chose is a bit misleading. I am just using it to indicate the problem though. You are thinking of 'doctors' as the doctors in our culture but I was hypothesising a situation in which the field is much more primitive. It is interesting to me to see how rationality is applied to a field where data is unavailable, scarce or inconclusive. I don't see the problem of saying 'I don't know' when the data is obviously insufficient for me to judge. I would actually consider it counter productive to give a probability in this case, as it might create the illusion that I know more than I actually do. In this sense my 'I don't know' is not a value of 0 but admitting that there is no reason to use rational jargon at all. The terms 'right hypothesis' and 'truth' are used, in the above comment, in the context of what is considered a good enough truth in science. You are right that this can be confusing if we get into the epistemological details though I thought it was sufficient for the purpose of communicating in this thread. I can change it to 'scientific fact' and we should be ok? Does that make sense?

0ChristianKl8y

After Popper science isn't about establishing truth or the right hypothesis. If you can do better than a random guess (the dart throwing monkey) than you have knowledge in the Bayesian sense. There could be situations where you really don't know more than the dart throwing monkey and were it thus doesn't make sense to speak about probability but in most cases we know at least a little bit.

0Erfeyah8y

I am not familiar with Popper but I would agree anyway. I will be more careful with my terms. Would 'scientific fact' work though? I think it does but I am open to being corrected. [1] What if a rational assessment of inconclusive data weighs you towards the wrong direction. Wouldn't you then start doing worse than the dart throwing monkey? I would challenge your 'in most cases' statement. I would also challenge the contention that a little bit is better than nothing according to [1].

0ChristianKl8y

No. Everything in science is falsifiable and open to challenge. It's certainly possible to be completely deceived by reality. Whenever you act where an outcome matters to you, you will take the expected outcomes into account. Even if you say "I don't know" you still have to make decisions about what you do about an issue. Maybe http://slatestarcodex.com/2013/08/06/on-first-looking-into-chapmans-pop-bayesianism/ is worth reading for you. The kind of "I don't know that you advocate is what Scott calls Anton-Wilsonism. When doing credence calibration I don't get result that indicate that I should label all claims 50/50.

0Erfeyah8y

I understand and agree with that. I am just trying to find the term I can use when discussing scientific results. I thought 'scientific fact' was ok cause it includes 'scientific' which implies all the rest. But yes the word 'fact' is misleading. Should we just call it 'scientific result'? What do you recommend? I can't stress enough how useful that link is to me as a new LW user. My criticisms are quite close to what David Chapman is saying and it is really nice to see how someone representative of LW responds to this. Discussing in LW is giving me the impression at the moment that I have to learn to talk in a new language. I have to admit that at the moment all the corrections you guys have indicated are an improvement on my previous way of expressing. Very exciting! But this is a great opportunity to deepen my understanding of the language by practising it. Let me try to reformulate my 'I don't know' in the Bayesian language. So, what I mean by 'I don't know' is that you should use a uniform distribution. For example, you have attached the label of 'Anton-Wilsonism' to me according to what I have currently expressed. I could assume, if you are literally using this way of thinking, that you went through the process of considering a weight for the probability that I am an exact match of what Scott is describing and decided that based on your current evidence I am. This also implies that you have, now or in the past, assigned ratings for all the assumptions and conclusions made in Scott's two paragraphs (there are quite a few) and you are applying all these to my model. So: 1. Did you really quantify all that or were these labels applied, as it commonly happens in humans, automatically? 2. Do you think that my recommendation of creating your model using uniform distributions would be useful as we are going through the process of getting more evidence about each other. It is a trivial, but indicative of an attitude, example that using my approach your action c

0ChristianKl8y

No, I don't think what you are saying is close to what Chapman is arguing. Chapman doesn't argue that we should say "I don't know" instead of pinning probability on statements where we have little knowledge. There are enough people who use terms like "scientific fact" without thinking in terms of falsificationism that it's not clear what's implied. To me your sentence sounds like you have a naive idea of what the word "truth" is supposed to mean. A meaning that you learned as child. There are some intuitions that come with that view of the world. Some of those intuitions will come into conflict if you come into contact with more refined ideas of what truth happens to be and how epistemology should work. There are various philosophers like Popper who have put forward more refined concepts. Eliezer Yudkowsky has put forward his own concepts on lesswrong. For binary yes/no predictions the uniform distribution leads to 50/50. https://www.metaculus.com, http://predictionbook.com/ and https://www.gjopen.com/ do have plenty of examples.

0Erfeyah8y

Sorry. I meant my general criticisms (which I haven't expressed), not in the sense of our current discussion. I wasn't very clear. I am not sure where you are getting that I "have a naive idea of what the word "truth" is supposed to mean.". Stating it is no justification. Pointing towards Popper or Yudkowsky is not justification either. You would need to take my statements that point towards my 'naivety' and deconstruct them so we can learn. I have from my side offered arguments and examples for the value of the 'I don't know' mentality and why it is useful but I feel you haven't engaged. I'm afraid I am not in a position to argue about this as I have only partially understood it. You can read here.

0ChristianKl8y

Chapman doesn't reject rationality but advocates transcending it. It's a different standpoint. You need to first adopt a framework to later transcend it. In Chapman view, LW type rationality is useful for people who move from Kegan 3 to Kegan 4. If you actually refined your concept of truth, there a good chance that you could point to philosophers that influenced it. This would allow me to address their arguments to the extent that I'm familiar with their notions of truth and how the relate to LW rationality. I see your argument as "But this isn't truth" without any deep argument of what you mean by "truth" or signs that you went through the process of refining a notion of what it means for yourself. You speak about "not fully believing" when the whole point of putting probabilities on the statements is that you don't fully know what's going to happen. There's the general mantra of "Strong opinions, loosely held." Starting a probability means that this is the likelihood that the information that's available in this moment warrants. It in no way implies that if other information is available in the next moment that the probability will stay the same. Constantly updating the probability as new information becomes available is part of the ideal of Bayesian rationality.

0Erfeyah8y

I do not reject rationality either. Why would I be here if I did so? I think you are misreading my contrarian approach as rejection. Talking in terms of references bears the danger of attaching labels to each other. It is much more accurate expanding on points. I respect if you don't have the time for that of course. Since you are asking though let's see where this comparison of readings take us. In terms of most Western philosophy I find the verbosity and tangle of self constructed concepts to be unbearable (though I have to clarify that the 2 pages of Popper that I read were perfectly clear). Wittgenstein's philosophical investigations is I believe a good antidote to a lot of the above. My study nowadays is focused on self observation, psychology, sociology, neuroscience as well as eastern philosophy ( I am not religious ). For example, you can find a perspective on truth by reading the full corpus of teaching stories/anecdotes of mullah Nasrudin. A deep definition? Truth is reality. It can not be reached or expressed through rationality but parts of it can be approximated/modelled in a way that can be useful. For discussing practical rationality though isn't it enough to say that truth is a belief that corresponds to reality? By this time I feel we have kind of lost the focus of our conversation. To recap, this was all about my comment on the possibility of rationally arrived beliefs being false due to the assessment of insufficient evidence. It was not meant as an attack on rationality but as constructive criticism and possible a way for me to be introduced to solutions.

0ChristianKl8y

That's evading the question. Or pretending that it doesn't exist. I can't spend time going deep into your argument when you use a naive definition of a term that can be stated in three words. Saying that there's a chance that it's false is no new issue but it's an issue that's already addressed. The whole point of putting a probability on a belief is that there's a chance that's false. It's not any new concern. Probability is made for not knowing whether a belief is true or false but having uncertainty about it..

0Erfeyah8y

Interesting. I guess your approach of just saying that you understand something but not expressing it in the discussion is not evasive at all. I will keep an open mind for future conversations because, believe it or not, I am open to learn from you, if there is something you can teach me. But at the moment you are just demonstrating a tendency towards 'name dropping' and 'name calling' which is not constructive at all.

0ChristianKl8y

I'm speaking about processes of reasoning. Saying "Truth is reality", tells me nothing about what it means for a probability to be true. It leaves everything important implicit. In the Kegan model that Chapman uses handling concepts this way happens at level three. The step to level four is to actually dig deeper and refine one's notions to be more specific. To use notions that come from a internally consistent system instead of being the naive notions of the concepts of the kind that people take up while being in high school. Chapman then goes and says: On the one hand I have the system of reasoning with probability and qualifiying my uncertainty with probability. On the other hand I also want to use predicate calculus and the system in which I can use predicate calculus is not the one that's ruled by Bayes rule. But that doesn't mean that one system is more true or more in line with reality. Both are ways to model the world.

0Erfeyah8y

Thank you for engaging :) For the next paragraphs I would like you to exercise humility and restrain your assessment of what I am saying until I have finished saying it. You can then assess it as a whole. My definition of truth was not three words. It was a small paragraph. Let me break it down: Now why is this useful? The first three words 'Truth is reality' acknowledges that there is existence and inevitably this X which you can call the world, nature, whole or reality (the one I used in this occasion as I found it cleaner of associations in our context) is inevitably equivalent to truth. Rational discussion, having as its basis the manipulation of symbols, is an abstraction of X and thus not X. Thus absolute truth is outside the realm of rationality. If you think this is what you describe as 'naive notions of the concepts of the kind that people take up while being in high school.' I can only say in my turn that your understanding seems naive to me. Here I state that although the first three words describe the deepest level of truth we can focus on truth as can be expressed in rational terms (mathematics are included in this) because it is demonstrably useful. But we should not confuse this truth with reality. We can call it 'relative truth' vs 'absolute truth' or truth vs Truth or whatever you fancy as long as we clarify our terms so we can talk. And here we are in the domain where I can learn from you. The domain of being efficient at using rationality. Notice that this sentence is a question where you can respond with a better conceptualisation. Rereading this sentence I already think I see its shortcomings. How about: 'For discussing practical rationality should we say that truth is a belief of which we can observe or demonstrate its relation to reality?'. Or should we just stop using the word truth for now? I am up for that. So to return to your statement: Indeed. That is why I did not move the conversation towards this direction, you did. I would inv

0ChristianKl8y

Let's take a statement like 2+2=4. It's not a statement about nature. It's a statement about how abstract mathematical entities relate to each other that are independent from nature. I can reason about whether certain statements about Hilberts hotel are true even through there's no such thing as Hilbert's hotel in reality. What you are saying looks to me like you didn't went through edge cases like this and decided whether you think statements surrounding Hilberts hotel shouldn't be called true. Going through edges cases leads to a refinement of concepts. It rather sounds to me like you think that those edge cases don't really matter and the intuitions you have of the concept of truth should count. You can't claim this and saying at the same time that someone's probability is false or wrong. You just defined truth in a way that it's an attribute of different claims. One way of dealing with a concept like this is to reference refined concepts like the concept of truth that Eliezer developed in the sequences. Dropping the concept (in LW speak tabooing it) is another way. If your objection to believing that "X happens with probability P" isn't anymore that this might be false, what's the objection about?

0Erfeyah8y

Before I continue with the discussion I have to say that this depth of analysis does not seem relevant to the practical applications of rationality that I asked about in the original post. The LessWrong wiki states under the entry for 'truth': This, 'relative truth' as I call it, would be perfectly adequate for our discussion before we started philosophising. Nevertheless, philosophising is good fun! :) So... I assume you are using 'nature' in the same way I use `reality'? If yes, it is absurd to say that these entities are independent of nature. Everything is part of nature. Nature is everything there is. It includes you. Your brain. Mathematics. The question you can ask is: Why does abstraction have these properties? Why do they sometimes describe other parts of reality? Does every mathematical truth have a correspondence to this other part of reality we call the physical world? These are all valid and fascinating questions. I clearly stated that 'relative truth' can approximate/model parts of 'absolute truth' in a way that can be useful. You definitely convinced me to be more careful when I use the word. Seriously. That is my objection. You think there is a conflict because you are not distinguishing between 'absolute' and 'relative' when you follow my definition. In the original post, I was just observing the situation we are in when we use rational assessment with incomplete data. I am interested to see if we can find ways to calibrate for such distortions. I will expand in future posts. Finally, I wouldn't want to give you the impression that I am certain about the view of 'truth' I am presenting here. And I hope you are not sure of your assessments either. But this is where I currently am. This is my belief system.

0ChristianKl8y

I didn't have the impression. Saying truth is about correspondence to reality is quite different than saying that it is about reality. In Bayesianism a probability that a person associates with an event should be a reflection of the information available to the person. Different people should come to the same probability when subject to exactly the same information but to the extent that different people are in the real world always exposed to different information it's subjective. In the frequentist idea of probability there's the notion that the probability is independent from the observer and the information that the observer has, but that assumption isn't there in the Bayesian notion of probability.

0Erfeyah8y

Right, and what my original post explores is that different people should come to the same inaccurate probability when subject to exactly the same incomplete information. Indeed this seemed to be something people are aware of from what I gathered from the answer of MrMind and this one from Vaniver. Vaniver in particular pointed me towards an attempt to model the issue in order to mitigate it but it presupposes a computable universe and, most importantly, that the agent has logical omniscience and an infinite amount of time. This puts it out of the realm of practical rationality. A brief description of further attempts to mitigate the issues left me, for now, unconvinced.

0ChristianKl8y

The idea of accuracy presupposes that you can compare a value to another reference. If I say that Alice is 1,60m tall but she's 1,65m, that's inaccurate. If I however say, that there's a 5% chance that Alice is taller than 1,60m that's not inaccurate. My ability to predict height might be badly calibrated and I have a bad Briers score or Log score.

0Erfeyah8y

I am using 'inaccurate' as equivalent to 'badly calibrated' here. Why do you feel it is important to make the distinction? I understand why it is important when dealing with clearly quantified data. But in every day life do you really mentally attempt to assign probability to all variables?

0ChristianKl8y

To determine whether a person is well calibrated or isn't you have to look at multiple predictions of the person. It's an attribute heuristic for decision making. On the other hand a single statement such as Alice is 1,60m might be inaccurate. Being inaccurate is a property of a statement and not just a property of how the statement was generated. Assigning probabilities to event takes effort. As such it's not something you can do for two-thousand statements in a day. To be able to assign probabilities it's also important to precisely define the belief. If I take a belief like "All people who clicked on 'Going' will come to the event tonight", I can assign a probability. The exercise of assigning that probability makes me think more clearly about the likelihood of it happening.

0Erfeyah8y

Thanks for the clarifications. One last question as I am sure all these will come out again and again as I am interacting with the community. Can you give me a concrete example of a complex, real life problem or decision where you used the assignment of probabilities to your beliefs to an extend that you find satisfactory for making the decision. I am curious to see the mental process of really using this way of thinking. I assume it is a process happening through sound in the imagination and more specifically through language (the internal dialogue). Could you reproduce it for me in writing?

0ChristianKl8y

I applied for a job. There was uncertainty around whether or not I get the job. Having an accurate view of the probability of getting the job informs the decision of how important it is to spend additional effort. I basically came up with a number and then ask myself whether I would be surprised if the event happens or doesn't happen. I currently don't have a more systematic process. ---------------------------------------- I remember a conversation with a CFAR trainer. I said "I think X is a key skill". They responded with: "I think it is likely that X is a key skill but I don't know that it has to be a key skill.". We didn't put numbers on it but having probabilities in the background results in us being able to discuss our disagreement even through we both think "X is likely a key skill". I had never someone outside of this community tell me "you are likely right but I don't see why I should believe that what you are saying is certain". The kind of mindset that produces a statement like this is about taking different probabilities seriously.

0Erfeyah8y

My thought is: 'I have reached this mindset through studying views of assumptions and beliefs from other sources. Maybe this is another way to make the realisation.' My doubt is: 'Maybe I am missing something that the use of probabilities adds to this realisation' Hope to continue the discussion in the future.

0ChristianKl8y

It's more than just a mindset. In this case the result were concrete discoursive practice. There are quite many people who profess to have a mindset that separates shades of gray. The amount of people who tell you voice disagreement when you tell them something they believe is likely to be true and that's important is much lower. Can you think of the last time where you cared about an issue and someone professed to believe what you likely believed to be true, that you disagreed with them? And stretch out the example?

0Erfeyah8y

Do I need to express it in numbers? In my mind I follow and practice, among others, the saying: “Study the assumptions behind your actions. Then study the assumptions behind your assumptions.” Having said that, I can not think of an example of applying that in a situation where I was in agreement. I am thinking that 'I would not be in agreement without a reason regarding a belief that I have examined' but I might be rationalising here. I will try to observe myself on that. Thanks!

0ChristianKl8y

We both had reasons for believing it to be true. On the other hand human believe things that are wrong. If you ask a Republican and a Democrat whether Trump is good for America they might have both reasons for their belief but they still disagree. That means for each of them there's a chance of them being wrong despite having reasons for their beliefs. The reasons he had in this mind pointed to the belief being true but they didn't provide him the certainty that it's true. It was a belief that was important enough for him to be right and not only have reasons for holding his belief. The practice of putting numbers on a belief forces you to be precise about what you believe. Let's say that you believe: "It's likely that Trump will get impeached." If Trump actually get's impeached you will tell yourself "I correctly predicted it, I was right". If he doesn't get impeached you are likely to think "When I said likely than it meant that there was a decent chance that he get's impeached but I didn't mean to say that the chance was more than 50%. The number forces precision. The practice of forcing yourself to be precise allows the development of more mental categories. When Elon Musk started SpaceX he reportedly thought that it had a 10% chance of success. Many people would think of 10% of success as. It's highly unlikely that the company succeeds. Elon on the other hand thought that given the high stakes 10% chance of success is enough to found SpaceX.

0Erfeyah8y

I will have to explore this further. At the moment the method seems to me to just give an illusion of precision which I am not sure is effective. I could say that I assign a 5% probability that the practice is useful to represent my belief. I will now keep interacting with the community and update my belief according to the evidence I see from people that are using it. Is this the right approach?

0ChristianKl8y

The word "useful" itself isn't precise and as such the precision of 5% might be more precise than warranted. Otherwise having your number and then updating it according to what you see from people using it, is the Bayesian way.

0Erfeyah8y

How would you express the belief?

0gjm8y

Sure. In other words, if you get fed bad enough data then you have (so to speak) anti-knowledge. Surely this isn't surprising?

0Erfeyah8y

No, not really surprising. I would just clarify though, that the data does not need to be 'bad' in the sense that it is false. We might have data that are accurate but misinterpret them by generalising to the larger context or mistakenly transposing them to a different one.

3MrMind8y

I don't think you're missing anything. Bayesian reasoning allows you to treat your data without introducing errors, but the results you come up with are a product of the available data and the prior model. This is a point that is often overlooked: if you start with a completely false model, even with Bayesian reasoning the data will get you further away from the truth (case in point: someone who believes in an invisible dragon which has to invent more and more complicated explanation for the lack of evidence). Bayesian probability is just the way of reasoning that introduces the least amount of error. To counter at least partially our fallibility, it's considered good practice to: * never put any assumption at precisely 0 or 1 probability; * leave always a reservoir of probability mass in your model to unknown unknows. Other than that, findind the truth is a quest that needs creativity, ingenuity and a good dose of luck.

1Douglas_Knight8y

A lot of people say that and go on to give a specific example, that Science did not accept general relativity on the basis that it explained the precession of Mercury, but rather on the novel prediction of solar lensing. But this is historically false. Scientists were much more impressed with the precession than the eclipse (even leaving aside Eddington's reliability).

0Vaniver8y

Note that experimental confirmation isn't really the thing here; experiments just give you data and the problem here is conceptual (the actual truth isn't in the hypothesis space). Most Bayes is "small world" Bayes, where you have conceptual and logical omniscience, which is possible only because of how small the problem is. "Big world" Bayes has universal priors that give you that conceptual omniscience. In order to make a real agent, you need a language of conceptual uncertainty, logical uncertainty, and naturalization. (Most of these frameworks are dualistic, in that they have a principled separation between agent and environment, but in fact real agents exist in their environments and use those environments to run themselves.)

0Erfeyah8y

I assume you are talking hypothetically and not really saying that we, in reality, have these priors? Is there an article about this 'small world' 'big world' distinction? This went completely over my head. Why did you bring agents in the conversation?

0Vaniver8y

I had in mind Solomonoff Induction. Here's the last time that came up; I think it's mostly in margins rather than an article on its own. Ah, because when talking about the how to model problems (which I think Bayesian rationality is an example of), agents are the things that do that.

0Erfeyah8y

Ok that makes sense. These approaches are trying to add considerations such as mine into the model. Not sure I see how that can solve the issue of "the truth missing from the hypothesis space". Or how accurate modelling of the agents can be achieved at our current level of understanding. Examples of real world applications instead of abstract formulations would be really helpful but I will study the article on Solomonoff induction.

0Vaniver8y

Solomonoff Induction contains every possible (computable) hypothesis; so long as you're in a computable universe (and have logical omniscience), the truth is in your hypothesis space. But this is sort of the trivial solution, because while it's guaranteed to have the right answer it had to bring in a truly staggering number of wrong answers to get it. It looks like what people do is notice when their models are being surprisingly bad, and then explicitly attempt to generate alternative models to expand their hypothesis space. (You can actually do this in a principled statistical way; you can track, for example, whether or not you would have converged to the right answer by now if the true answer were in your hypothesis space, and call for a halt when it becomes sufficiently unlikely.) Most of the immediate examples that jump to mind are mathematical, but that probably doesn't count as concrete. If you have a doctor trying to treat patients, they might suspect that if they actually had the right set of possible conditions, they would be able to apply a short flowchart to determine the correct treatment, apply it, and then the issues would be resolved. And so when building that flowchart (i.e. the hypothesis space of what conditions the patient might have), they'll notice when they find too many patients who aren't getting better, or when it's surprisingly difficult to classify patients. If people with disease A cough and don't have headaches, and people with disease B have headaches and don't cough, on observing a patient who both coughs and has a headache the doctor might think "hmm, I probably need to make a new cluster" instead of "Ah, someone with both A and B."

0Erfeyah8y

I read the article and I have to say that the approach is fascinating in its scale and vision. And I can see how it might lead to interesting applications in computer science. But, in its current state, as an algorithm for a human mind.. I have to admit that I can not justify investing the time for even attempting to apply it. Thank you for all the info! :)

0MaryCh8y

Seems like under that system, some headaches would just go away on their own, without symptoms of cold appearing later, which would mean the probability of cancer goes up. The patient panics, gets checked for cancer (since a recurring headache would hardly be the only symptom), no such disease is found, so there apparently has to be a new category for non-cancer, non-cold headaches. So usually, the evidence soup has pieces that don't conform to a theory, and if the stakes are high enough, people will go looking for them and maybe use the BR for it.

0Erfeyah8y

Yes, the example is a bit misleading. My purpose is to observe rationality in a field where data is unavailable, scarce or inconclusive. So use the example as a loose analogy for the purpose of communicating the point.

1MaryCh8y

[I think that] where data are unavailable, you cannot really constrain your expectations of them. You can build models, even, perhaps, assign weights within them, but there is no basis to choose between one model and another, and every number of models you find 'satisfying' remains arbitrary. (This is why I don't argue about religion, for example.)

0AmagicalFishy8y

I don't think you're missing anything, no.

There are fields of science where different research teams use not-quite-the-same units of measurement. For example, in phytohormonology, the amount of a hormone in plant tissue can be expressed in (nano)grams per grams of dry weight or fresh weight, and people who compare values expressed in different ways cringe because they understand how inexact it really is. There simply is no reliable scale that would allow recalculation, especially for different species etc. (not to mention all the other ways in which drawing conclusions from only the amount of such... (read more)

0ChristianKl8y

Standardization is the core word. Institutions like ISO exist to create common standards. Jounals can then force scientists to actually follow standards. Controlled vocabularies and applied ontology seem other key words. How useful standardization happens to be depends a lot on the quality of the standard. The DSM-V for example seems to be a standard that holds science back and as a result there are calls for funding research that tries to use new standards.

0MaryCh8y

Thank you! I'll look at/read "Applied ontology: an introduction" (ed. by Munn and Smith) - the results are rather varied and have their own developed terminology, and this one looks as good place to start as any. Edit to add: tentatively, the "automated information systems" angle might not be what I'm looking for:(

0ChristianKl8y

Automated information system require fixed vocabulary. If people who observe rats have a different idea about what a leg happens to be then the people who study humans (the leg is the part between the foot and the knee) there are problems with translating knowledge. Humans might be smart enough to do the translation but computers won't. As a result there's interest in standardization. Bioinformatics needs the standardization and that's where Barry Smith comes from. Bioinformatics has the interests in standardization because automated information systems don't work without it. ---------------------------------------- I remember a story, which I think cames from People Works (a book about Google's HR department). It made the point that it's not trivial to have a charged definition in a company of what it means to have 10 employees. The people who pay the wages might count 6 full time employees plus 4 half-time employees as 8 employees. When it comes to paying health insurance, it's 10 employees. The HR department might count prospective employees as an employee the moment the employee signs the offer while another department waits till their starting date. The fact that Google has a charged definition of employee allowed them to do much better statistics.

0MaryCh8y

Yes, I appreciate the effect of automation on standardization, it is really a great thing. I just have the impression that differences stemming from the deep, very much method-shaped variability of research - like 'radiation' in the evolutionary sense - might be only superficially addressed using only the standardization-as-I-have-read-of-it. (I'm still reading, and expect this to change.) I'm starting with an image of 'variable and its measure can be only tenuously linked (like 'length' and 'meter) to pretty much baked together (like the phytohormone example)'. This image might itself be just wrong.

0ChristianKl8y

Meter is not our only measure of length. We also have astronomical units to measure length. In school they taught us that they were different units and that we can speak with more precision about the distance between two stars if we talk in astronomical units. For a long time there were also a bunch of interesting questions such as whether it makes sense to say that the norm meter in Paris is 1-meter long and whether it stays exactly 1-meter long is it's surface oxidates a bit. Metal changes it's length at different temperature. That means you need a definition of temperature to define the length of the meter if you define it over the norm meter. Newton thought that there's was a fixed "temperature of blood". Fahrenheit used "body temperature" as a measuring stick for a specific temperature. It took a lot of science to find the freezing point and the boiling point of water as the perfect way to norm temperature. If you shape the vessel the right way, it's possible to boil water at 102 degree C, so they needed to specify the right conditions.

0MaryCh8y

I either didn't know or hadn't thought in the context about most of what you say here, thank you. Yet this (the exact length of a meter) is more-or-less settled, in the sense that very many people use it without significant loss of what they want to convey. This is kind of exactly the thing I'd like to learn about - how unit-variable relationships evolve and come to some 'resting position'. How people first come to think about the matter of a subject, than about the ways to describe it, and finally about the number of a common 'piece' used to measure it.

0ChristianKl8y

I think the Applied Ontology book is worth reading as it touches a lot of the practical concerns that come with the need for standardization due to automated knowledge processing. Even if you aren't interested in automated knowledge processing it still useful. Inventing Temperature: Measurement and Scientific Progress by Hasok Chang is a good case study for how our measure of temperature evolved. Temperature is a good example because conceptualizing it is harder than conceptualizing length. In the middle ages people had their measures for length but they didn't have one for temperature. The definition of the meter over the wavelength of light instead of over the norm meter was settled in 1960 but the amount of people for whom there were practical concerns was relatively little. Interestingly we have at the moment a proposed change to the SI system that redefines the kilogram: https://en.wikipedia.org/wiki/Proposed_redefinition_of_SI_base_units It changes the uncertainity that we have over a few constants. Beforehand we had an exact definition of the kilogram and afterwards we only know 8 digits of accuracy. On the other hand we get more accuracy for a bunch of other measurements. It might be worth reading a bit into the debate if you care about how standards are set.

How does a rational actor resolve the emperor's clothes?

Story link: http://www.andersen.sdu.dk/vaerk/hersholt/TheEmperorsNewClothes_e.html

Specifically, insert ourselves into every step of the process.

1) You're the emperor. Two tailors come to you saying they can make you a suit that cannot be seen by those that are stupid and/or unfit for their current position.

Answer to this, I think, is: You don't believe this magical stuff, see it for the scam that it is and tell them to bugger off.

2) You as the emperor, somehow agree to this. They take your measurem... (read more)

3Viliam8y

Collect more evidence. If possible, find a person who never heard about the supposed properties of the clothes, and ask them to describe them to you. If they can't, maybe they are stupid, but then find another one and... uhm, if everyone who didn't hear about the supposed properties of the clothes is stupid, that's suspicious. Unexpectedly invite a few painters, put them in different corners of the room, and ask them to paint you in the clothes. Alternatively, test your ministers. First, meet them with the clothes; next, without them. If they see the clothes both times... Put the clothes in a bag. Add a few empty bags. Ask your ministers which bag contains the clothes. If all of them failed, ask the tailors.

2MockTurtle8y

Interesting questions to think about. Seeing if everyone independently describes the clothes the same way (as suggested by others) might work, unless the information is leaked. Personally, my mind went straight to the physics of the thing, 'going all science on it' as you say - as emperor, I'd claim that the clothes should have some minimum strength, lest I rip them the moment I put them on. If a piece of the fabric, stretched by the two tailors, can at least support the weight of my hand (or some other light object if you're not too paranoid about the tailor's abilities as illusionists), then it should be suitable. Then, when your hand (or whatever) goes straight through, either they'll admit that the clothes aren't real, or they'll come up with some excuse about the cloth being so fine that it ripped or things go straight through, at which point you can say that these clothes are useless to you if they'll rip at the slightest movement or somehow phase through flesh, etc. Incidentally, that's one of my approaches to other things invisible to me that others believe in. Does it have practical uses or create a physical effect in the world? If not, then even if it's really there, there's not much point in acknowledging it...

0ChristianKl8y

The irony of the situation is that some fancy closes that are today worn in Milan leave a large part of the person naked. As different people for the color of the clothes and for more details. If the people really can see the clothes they should be able to describe the clothes in the same way. If there already common knowledge about the color of the clothes or details then it would be required to see the clothes in a new context. How do the clothes look like when they get wet? If two people agree how the clothes look under a completely new context it's more likely that they don't just tell you the answer they learned by heart.

[-]James_Miller8y20

If I email someone non-famous to be on my podcast and they don't respond should I take that as a "no" or as a "didn't get the message try again".

8gjm8y

I make no claim to know what you should do, but what I would do in that situation is: wait at least a month; then email them again saying: no obligation to reply or anything, just wanted to make sure emails aren't going astray. If still no reply, assume that either they aren't interested or they're spam-binning your emails and you'll never reach them by email anyway.

[-]hodgestar8y20

Less Wrong has a number of participants who endorse the idea of assigning probability values to beliefs. Less Wrong also seems to have a number of participants who broadly fall into the "New Atheist" group, many of the members of which insist that there is an important semantic distinction to be made between "lack of belief in God" and "belief that God does not exist."

I'm not sure how to translate this distinction into probabilistic terms, assuming it is possible to do so-- it is a basic theorem in standard probability theo... (read more)

4Erfeyah8y

Shouldn't a lack of belief in god imply: P(not("God exists")) = 0.5 P("God exists") = 0.5 (I am completely ignoring the very important part of defining God in the sentence as I take the question to be asking of a way to express 'not knowing' in probabilistic terms. This can be applied to any subject really.)

0hairyfigment8y

That is indeed the chief problem here. I'm assuming you're talking about the prior probability which we have before looking at the evidence.

0Lumifer8y

No. Why would it? "I don't know" is a perfectly valid answer. Sometimes it's called Knightian uncertainty or Rumsfeldian "unknown unknowns".

2Erfeyah8y

I agree that "I don't know" is a better answer as there is no reason to talk in rational jargon in a case like this. Especially when we haven't even defined our terms. But could you explain to me why assigning 0.5 probabilities in the two opposites (assuming the question is clear and binary) does not make sense for expressing ignorance?

0Lumifer8y

Because you lose the capability to distinguish between the "I know the probabilities involved and they are 50% for X and 50% for Y" and "I don't know". Look at the distributions of your probability estimates. For the "I don't know" case it's a uniform distribution on the 0 to 1 range. For the "I know it's 50%" it's a narrow spike at 0.5. These are very different things.

0Erfeyah8y

Ahhh, this is confusing me. I intuitively feel a 50-50 chance implies a uniform distribution. But what you are saying about the distribution being a spike for 0.5 makes total sense. Well, I guess I have a bit of studying to do...

3gjm8y

Being a full-on Bayesian means not only having probability assignments for every proposition, but also having the conditional probabilities that will allow you to make appropriate updates to your probability assignments when new information comes in. The difference between "The probability of X is definitely 0.5" and "The probability of X is somewhere between 0 and 1, and I have no idea at all where" lies in how you will adjust your estimates for Pr(X) as new information comes in. If your estimate is based on a lot of strong evidence, then your conditional probabilities for X given modest quantities of new evidence will still be close to 0.5. If your estimate is a mere seat-of-the-pants guess, then your conditional probabilities for X given modest quantities of new evidence will be all over the place. Sometimes this is described in terms of your probability estimates for your probability estimates. That's appropriate when, e.g., what you know about X is that it is governed by some sort of random process that makes X happen with a particular probability (a coin toss, say) but you are uncertain about the details of that random process (e.g., does something about the coin or how it's tossed mean that Pr(heads) is far from 0.5?). But similar issues arise in different cases where there's nothing going on that could reasonably be called a random process but your degree of knowledge is greater or less, and I'm not sure the "probabilities of probabilities" perspective is particularly helpful there.

0Erfeyah8y

Thanks for the detailed explanation. It helps!

0Lumifer8y

Well, imagine a bet on a fair coin flip. That's a 50-50 chance, right? And yet there is no uniform distribution in sight.

0niceguyanon8y

So if we can distinguish between Could we further distinguish between Let's say a biased coin with unknown probability of landing heads is tossed, p is uniform on (0,1) and "I don't know" means you can't predict better than randomly guessing. So saying p is 50% doesn't matter because it doesn't beat random. But what if we tossed the coin twice, and I had you guess twice, before the tosses. If you get at least one guess correct then you get to keep your life. Assuming you want to play to keep your life, then how would you play? Coin is still p uniform on (0,1), but it seems like "I don't know" doesn't mean the same thing anymore, because you can play in a way that can better predict the outcome of keeping your life. You would guess (H,T) or (T,H) but avoid randomly guessing because it would produce things like (H,H) which is really bad because if p is uniform on (0,1), then probability of heads is 90% is just as likely as probability of heads is 10%, but heads at 10% is really bad for (H,H), so bad that even 90% heads doesn't really help that much more. If p is 90% or 10%, guessing (H,T) or (T,H) would result in the same small probability of dying at 9%. But (H,H) would result in at best 1% or 81% chance of dying. Saying I don't know in this scenario doesn't feel the same as I don't know in the first scenario. I am probably confused.

0Lumifer8y

But you've changed things :-) In your situation you know a very important thing: that the probability p is the same for both throws. That is useful information which allows you to do some probability math (specifically compare 1 - p(1-p) and 1 - p^2). But let's say you don't toss the same coin twice, but you toss two different coins. Does guessing (H,T) help now?

0niceguyanon8y

I understand now. Thanks!

0Erfeyah8y

You are obviously right! This is helpful :) Now, just to make sure I got it, does this make sense: the question of gods existence (assuming the term was perfectly defined) is a yes/no question but you are conceptualising the probability that a yes or a no is true. That is why you are using a uniform distribution in a question with a binary answer. It is not representing the answer but your current confidence. Right?

1Lumifer8y

Skipping a complicated discussion about many meanings of "probability", yes. Think about it this way. Someone gives you a box and says that if you press a button, the box will show you either a dragon head or a dragon tail. That's all the information you have. What's the probability of the box showing you a head if you press the button? You don't know. This means you need an estimate. If you're forced to produce a single-number estimate (a "point estimate") it will be 50%. However if you can produce this estimate as a distribution, it will be uniform from 0 to 1. Basically, you are very unsure about your estimate. Now, let's say you had this box for a while and pressed the button a couple thousands of times. Your tally is 1017 heads and 983 tails. What is your point estimate now? More or less the same, rounding to 50%. But the distribution is very different now. You are much more confident about your estimate. Your probability estimate is basically a forecast of what do you think will happen when you press the button. Like with any forecast, there is a confidence interval around it. It can be wide or it can be narrow.

2Vaniver8y

You're right about the probabilistic statements, with a potentially tangential elaboration. There are nonsense sentences--not contradictions ("A and not A") but things that fail to parse ("A and")--and it doesn't make sense to assign probabilities to those. One might claim that "God exists" is a nonsense sentence in that way, but I think most New Atheists don't take that approach. The distinction that people are drawing is basically which framing should have the benefit of the doubt, since not believing a new statement is the default. This is much more important for social rationality / human psychology than it is for Bayesianism, where you just assign a prior and then start calculating likelihood ratios.

0ChristianKl8y

From my perspective, a belief needs to be about empiric facts to have a probability attached to it. I need to be able to clearly describe how the belief could be tested in principle. In addition to beliefs about empiric facts there are also beliefs like "Nobody loves me." that aren't about specific empiric facts but that still matter a great deal.

0MrMind8y

I cannot speak for other atheists, but as far as I'm concerned I agree with you. Since we have a hard time defining even a human being, I accept that "God" cannot be clearly defined in any model, but I accept that there are narrations that points to some being of divine nature, and I accept that as a valid 'reference' to God(s). To those, I give a very low probability, with very little Knightian uncertainty (meaning that I also give very little probability to future evidence that would raise this probability, and high probability to evidence that will lower this value). For that account of the divine, I consider myself a full fledged atheist. There are other narrations though, and I've heard of definitions that basically reduce to "the law of physics", to which I give obviously very high probability with very high meta-certainty. There might be definitions or narrations that are in the middle, though. On that account, I cannot say precisely what my probabilities are, and thus would be appropriate to say that I lack a belief in this kind of god, more than a definite belief in the non-existence.

0gjm8y

It seems to me that someone could quite consistently hold the following position:

[-]madhatter8y20

Thanks for this topic! Stupid questions are my specialty, for better or worse.

1) Isn't cryonics extremely selfish? I mean, couldn't the money spent on cryopreserving oneself be better spend on, say, AI safety research?

2) Would the human race be eradicated if there is a worst-possible-scenario nuclear incident? Or merely a lot of people?

3) Is the study linking nut consumption to longevity found in the link below convincing?

http://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2173094

And if so, is it worth a lot of effort promoting nut consumption in moderation?

2Thomas8y

Here comes another "stupid question" from this one. Couldn't the money spent on AI safety research be better spend on, say, AI research?

5Vaniver8y

There's something like 100 times as much funding for AI research as there is for AI safety research. In general, it seems like it would be weird to have only 1% of the effort in a project spent on making sure the project is doing the thing that it should be doing. For this specific question, I like Stuart Russell's approach:

2MrMind8y

Well, the whole point of this forum is to convince someone that the answer is most definitely not.

3gjm8y

It really isn't. One of the reasons for the founding of this forum, yes. But what this forum is meant to be for is advancing the art of human rationality. If compelling evidence comes along that AI safety research is useless and AI research is vanishingly unlikely to have the sort of terrible consequences feared by the likes of MIRI, then "this forum" should be very much in the business of advocating against AI safety research.

2VincentYu8y

In support of your point, MIRI itself changed (in the opposite direction) from its former stance on AI research. You've been around long enough to know this, but for others: The former ambition of MIRI in the early 2000s—back when it was called the SIAI—was to create artificial superintelligence, but that ambition changed to ensuring AI friendliness after considering the "terrible consequences [now] feared by the likes of MIRI". In the words of Zack_M_Davis 6 years ago: I've always thought it's a shame they picked the name MIRI over SIFAAIDWSBBIUDC.

2Lumifer8y

Or maybe because SIAI realized their ability to actually create an AI is non-existent

0MrMind8y

Ha! It's wonderful news that you can take it off! For me you're the closest human (?) correlate to the man with the hat from XKCD, and I mean that as a compliment.

0Lumifer8y

I take it as such :-) You do mean the black hat guy, right? (there is also a white hat guy who doesn't pop up as frequently).

0MrMind8y

Yes, the black hatter. I totally forgot about the white hat guy...

0MrMind8y

You're right, but. The whole story goes like this: Eliezer founded this forum to advancing the art of human rationality, so that people would stop making silly objections to the issue of AI safety like "intelligence would surely bring about morality" and things like that. The focus of LW is human rationality and of MIRI is AI safety, but as far as I can tell, we still haven't found any valid objections to the orthogonality thesis. On the contrary, the issue of autonomous agents safety is gaining traction and recognition. I do agree that if we found a strong objections we should change perspective, but we still haven't and indeed we are seeing more and more worrisome examples.

0Thomas8y

I know that. But the whole point of this thread is to ask stupid questions, isn't it? And sometimes apparently the stupidest question, isn't stupid after all.

0Lumifer8y

Yes.

0Viliam8y

If we are talking about "extremes", what is the base set here: people's usual spending habits? Because I don't think cryonics is more selfish than e.g. buying an expensive car.

0MrMind8y

Well, 'better' here does all the work. It depends on your model and ethics: for example if you think that resuscitation is probably nearer then full AGI, then it's better to be frozen. This question I couldn't parse correctly. A nuclear war is improbable to wipe out humanity in its entirety, whereby a lot of people is th exact opposite of extinction, so...? This is far from a stupid question. The sample sizes are at least large, but it has the usual problem of using p-values, which are notoriously very fragile. It would require someone acquainted with statistics to judge better the thing, if it can be done at all.

[-]Bound_up8y00

I'm looking for a link I saw on Slate Star Codex. It was poetry written by a woman who took drugs every day for a year (something like that). Anyone know where I might find it?

0michaelkeenan8y

That sounds like Aella, who wrote about taking acid every week for a year. Here's her reddit post about it; it includes some art she made, and one poem.

[-]ChristianKl8y00

How many jobs that were done 50 years ago still exist in roughly the same form?

[-]VincentYu8y00

Why is downvoting disabled, for how long has it been like this, and when will it be back?

The original purpose of downvoting was to allow community moderation. Here, "moderation" means two things: (1) Giving higher visibility to high-quality content. This functionality we still have, it's the upvotes. (2) Removing low-quality content. Comments with karma below -5 and their whole subthreads are collapsed by default. This is especially important when some newcomers start spamming LW with a lot of low-quality comments. It happened more frequently in the past when LW was more popular.

And the "community" aspect means that these decisions about what to show prominently and what to hide are done by the local "hive mind", i.e. everyone, more precisely anyone above some amount of karma. This is good for several reasons: "wisdom of the crowds", preventing a few people from getting disproportional power, but most practically because moderators are busy and unable to review everything.

Why was it disabled:

The previous political debates on LW attracted one very persistent and very "mind-killed" person, known as Eugine. This guy made it his personal mission to promote neoreactionary politics on LW, and to harass away everyone who disag... (read more)

2VincentYu8y

Thanks for writing such a comprehensive explanation!

0Erfeyah8y

I am very new here but my impression from reading around is that people were taking advantage of the system by creating multiple accounts and downvoting comments that opposed them in order to appear to be right. I am not sure though.

0Viliam8y

Correct, with the addition that it was only one person. Very persistent, though... keeps doing this for years.

More from Erfeyah

Curated and popular this week