= 27d567bc2d48d9f40e9ccc20ea370b15)
private_messaging comments on Rationality Quotes September 2013 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (456)
Well, I too expect some form of something that we would call "AI", before 2100. I can even buy into some form of accelerating progress, albeit the progress would be accelerating before the "AI" due to the tools using relevant technologies, and would not have that sharp of a break. I even do agree that there is a certain level of risk involved in all the future progress including progress of the software.
I have a sense you misunderstood me. I picture this parallel world where legitimate, rational inferences about the AI risk exist, and where this risk is worth working at in 2013 and stands out among the other risks, as well as any other pre-requisites for making MIRI worthwhile hold. And in this imaginary world, I expect massively larger support than "Steven Hawkins hooked up with FHI" or what ever you are outlining here.
You do frequently lament that the AI risk is underfunded, under-supported, and there's under-awareness about it. In the hypothetical world, this is not the case and you can only lament that the rational spending should be 2 billions rather than 1 billion.
edit: and of course, my true rejection is that I do not actually see rational inferences leading there. The imaginary world stuff is just a side-note to explain how non-experts generally look at it.
edit2: and I have nothing against FHI's existence and their work. I don't think they are very useful, or address any actual safety issues which may arise, though, but with them I am fairly certain they aren't doing any harm either (Or at least, the possible harm would be very small). Promoting the idea that AI is possible within 100 years, however, is something that increases funding for AI all across the board.
Right, this just goes back to the same disagreement in our models I was trying to address earlier by making predictions. Let me try something else, then. Here are some relevant parts of my model:
Luke, why are you arguing with Dmytry?
The question should not be about "highly credentialed" people alone, but about how they fare compared to people who are rather very low "credentialed".
In particular, on your list, I expect people with fairly low credentials to fare much worse, especially at identification of the important issues as well as on rational thinking. Those combine multiplicatively, making it exceedingly unlikely - despite the greater numbers of the credential-less masses - that people who lead the work on an important issue would have low credentials.
What's EA? Effective altruism? If it's an existential risk, it kills everyone, selfishness suffices just fine.
Ohh, come on. That is in no way a demonstration that insurance companies in general follow faulty strategies, and especially is not a demonstration that you could do better.
Indeed.
A selfish person protecting against existential risk builds a bunker and stocks it with sixty years of foodstuffs. That doesn't exactly help much.
The quality of life in a bunker is really damn low. Not to mention that you presumably won't survive this particular risk in a bunker.
For what existential risks is this actually an effective strategy?
A global pandemic that kills everyone?
No doubt! I wasn't comparing highly credentialed people to low-credentialed people in general. I was comparing highly credentialed people to Bostrom, Yudkowsky, Shulman, etc.
But why exactly would you expect conventional researchers in AI and related technologies (also including provable software, as used in the aerospace industry, and a bunch of other topics), with credentials and/or accomplishments in said fields, to fare worse on that list's score?
Furthermore, with regards to the rationality, risks of mistake, and such... very little was done that can be checked for correctness in a clear cut way - most is of such nature that even when wrong it would not be possible to conclusively demonstrate it wrong. The few things that can be checked... look, when you write an article like this , discussing irrationality of Enrico Fermi, there's a substantial risk of appearing highly arrogant (and irrational) if you get the technical details wrong. It is a miniature version of AI risk problem - you need to understand the subject, and if you don't, there's negative consequences. It is much, much easier to not goof up in things like that, than AI direction.
As you guys are researching into actual AI technologies, the issue is that one should be able to deem your effort less of a risk. Mere "we are trying to avoid risk and we think they don't" can't do. The cost of a particularly bad friendly AI goof-up is a sadistic AI (to borrow the term from Omohundro). A sadistic AI can probably run far more tortured minds than a friendly AI can run minds, by a very huge factor, so the risk of a goof up must be quite a lot lower than anyone demonstrated.
BTW, I went back and numbered the items in my list so they're easier to refer to.
Because very few people in general, including credentialed AI people, satisfy (1), (2), (3), (5), (6), (7)†, (8), (10), and (12), but Bostrom, Yudkowsky and Shulman rather uncontroversially do satisfy those items. I also expect B/Y/S to outperform most credentialed experts on (4), (9), and (11), but I understand that's a subjective judgment call and it would take a long time for me to communicate my reasons.
† The AI risk part of 7, anyway. Obviously, AI people specifically know a lot about AI.
Edit: Also, I'll briefly mention that I haven't downvoted any of your comments in this conversation.
Ok, let's go over your list, for the AI people.
If EA is effective altruism, that's not relevant because one doesn't have to be an altruist to care about existential risks.
I expect them to be able to come up with that independently if it is a good idea.
I expect intelligent people to be able to foresee risks, especially when prompted by the cultural baggage (modern variations on the theme of Golem)
Well, that ought to imply some generally better ability to evaluate hard to calculate probabilities, which would imply that you guys should be able to make quite a bit of money.
The question is how well are they calibrated, not how much time they spent. You guys see miscalibration of famous people everywhere, even in Enrico Fermi.
Once again, how unbiased is what's important, not how much time spent on a very specific way to acquire an ability. I expect most accomplished people to have encountered far more feedback on being right / being wrong through their education and experience.
Doesn't apply to people in AI related professions.
The way to raise VoI is prior history of thinking about something else for a living, with impressive results.
Well, less credentialed people are just like this except they don't have a laboratory inside of which they are sane, that's usually why they are less credentialed in the first place.
Of your 3, I only weakly expect Bostrom to have learned the necessary fundamentals for actually applying Bayes theorem correctly in somewhat non-straightforward cases.
Yes, the basic formula is simple, but derivations are subtle and complex for non independent evidence or cases involving loops in the graph or all those other things...
It's like arguing that you are better equipped for a job at Weta Digital than any employee there because you know quantum electrodynamics (the fundamentals of light propagation), and they're using geometrical optics.
I expect many AI researchers to understand the relevant mathematics a lot, lot better than the 3 on your list.
And I expect credentialed people in general to have a good understanding of the variety of derivative tricks that are used to obtain effective results under uncertainty when the Bayes theorem can not be effectively applied.
Yeah, well, and I expect non-credentialed people to have too much to lose from backing out of it in the event that the studies return a negative.
You lose me here.
I would make a different list, anyway. There's my list:
Relevant expertise as measured by educational credentials and/or accomplishments. Expertise is required for correctly recognizing risks (e.g. an astronomer is better equipped for recognizing risks from the outer space, a physicist for recognizing faults in a nuclear power plant design, et cetera)
Proven ability to make correct inferences (largely required for 1).
Self preservation (most of us have it)
Lack of 1 is an automatic dis-qualifier in my list. It doesn't matter how much you are into things that you think are important for identifying, say, faults in a nuclear power plant design. If you are not an engineer, a physicist, or the like, you aren't going to qualify for that job via some list you make yourself, which conveniently omits (1).
edit: list copy paste failed.
I disagree with many of your points, but I don't have time to reply to all that, so to avoid being logically rude I'll at least reply to what seems to be your central point, about "relevant expertise as measured by educational credentials and/or accomplishments."
Who has educational credentials and/or accomplishments relevant to future AGI designs or long-term tech forecasting? Also, do you particularly disagree with what I wrote in AGI Impact Experts and Friendly AI Experts?
Also, in general, I'll just remind everyone reading this that I don't think these meta-level debates about proper social epistemology are as productive as object-level debates about strategically relevant facts (e.g. facts relevant to the theses in this post). Argument screens off authority, and all that.
Edit: Also, my view of Holden Karnofsky might be illustrative. I take Holden Karnofsky more seriously than almost anyone on the cost-effectiveness of global health interventions, despite the fact that he has 0 relevant degrees, 0 papers published in relevant journals, 0 awards for global health work, etc. Degrees and papers and so on are only proxy variables for what we really care about, and are easily screened off by more relevant variables, both for the case of Karnofsky on global health and for the case of Bostrom, Yudkowsky, Shulman, etc. on AI risk.
For Karnofsky and to some extent Bostrom yes, Shulman is debatable, Yudkowsky tried to get screened (tried to write a programming language, for example, wrote a lot of articles on various topics, many of them wrong, tried to write technical papers (TDT), really badly), and failed to pass the screening by a very big margin. Entirely irrational arguments about 10% counter-factual impact of his are also a part of failure. Omohundro passed with flying colours (his PhD is almost entirely irrelevant at that point, as it is screened off by his accomplishments in AI).
There's the more relevant accomplishments, there are less relevant accomplishments, and lacks of accomplishment.
I agree that a discussion of strategically relevant facts would be much more productive. I don't see facts here. I see many speculations. I see a lot of making things up to fit the conclusion.
If I were to tell you that I can, for example, win a very high stakes programming contest (with a difficult, open problem that has many potential solutions that can be ranked in terms of quality), the discussion of my approach to the contest problem between you and me would be almost useless for your or my prediction of victory (provided that basic standards of competence are met), irrespective of whenever my idea is good. Prior track record, on the other hand, would be a good predictor. This is how it is for a very well defined problem. It is not going to be better for a less well understood problem.
Exactly. All of this is wasted effort once either FAI or UFAI is developed.
'EA' here refers to the traits a specific community seems to exemplify (though those traits may occur outside the community). So more may be suggested than the words 'effective' and 'altruism' contain.
In terms of the terms, I think 'altruism' here is supposed to be an inclination to behave a certain way, not an other-privileging taste or ideology. Think 'reciprocal altruism'. You can be an egoist who's an EA, provided your selfish calculation has led you to the conclusion that you should devote yourself to efficiently funneling money to the world's poorest, efficiently reducing existential risks, etc. I'm guessing by 'EA' Luke has in mind a set of habits of looking at existential risks that 'Effective Altruists' tend to exemplify, e.g., quantifying uncertainty, quantifying benefit, strongly attending to quantitative differences, trying strongly to correct for a specific set of biases (absurdity bias, status quo bias, optimism biases, availability biases), relying heavily on published evidence, scrutinizing the methodology and interpretation of published evidence....
My own experience is that I independently came up with a lot of arguments from the Sequences, but didn't take them sufficiently seriously, push them hard enough, or examine them in enough detail. There seems to be a big gap between coming up with an abstract argument for something while you're humming in the shower, and actually living your life in a way that's consistent with your believing the argument is sound.
But we are speaking of credentialed people. They're fairly driven.
Furthermore, general non acceptance of an idea is evidence that the idea is not good. You can't seriously be listing general non acceptance of your ideas by the relevant experts as the reason why you are superior to those experts, because same non acceptance lowers the probability that those ideas are correct, proportionally to how much it raises how exceptional you are for holding those views. (The biggest problem with "Bayesianism" is dis-balanced/selective updates)
In particular, when it comes to the interview that he linked for reasons why value the future...
First off, if one can support existential risk for non Pascal's wager type reasons then enormous utility of the future should not be relevant. If it is actually a requirement then I don't think there's anything to discuss here.
Secondarily, the most common norm of morality (Assuming we ignore things like Sharia), as specified in the laws of progressive countries, or as extrapolation of legal progress in less progressive ones, is to value the future people (we disapprove of smoking while pregnant), but not value counter-factual creation of future people (we allow abortion, and especially when the child would be disadvantaged and not have a fair chance). Rather than inferring the prevailing morality from the law and discussing it, various bad ideas are invented and discussed to make the argument appear stronger than it really is.
It is not that I am not exposed to this worldview. I am. It is that choosing between A: hurt someone, but a large number of happy people will be created, and B: not hurt someone, but a large number of happy people will not be created (with the deliberate choice having the causal impact on the hurting and creation), A is both illegal and immoral.
When I hear that Joe has a new argument against a belief of mine, then my confidence in my belief lowers a bit, and my confidence in Joe's competence also lowers a bit. If I then go on to actually evaluate the argument in detail and discover that it's an extraordinarily poor one, this should generally increase my confidence to higher than it was before I heard that Joe had an argument, and it should further lower my confidence in Joe's competence.
I've spent enough time looking at the specific arguments for and against many of these propositions to have the contents of those arguments overwhelm my expertise priors in both directions, such that I just don't see a whole lot of value in discussing anything but the arguments themselves, when my goal (and yours) is to figure out the level of merit of the arguments.
It sounds like you're committing the Pascal's Wager Fallacy Fallacy. If you aren't, then I'm not understanding your point. Large future utilities should count more than small future utilities, and multiplying by low probabilities is fine if the probabilities aren't vanishingly low.
I think there's a quantitative tradeoff between the happiness of currently existent people and the happiness of possibly-created people. A strict rule 'Counterfactual People Have Absolutely No Value' leads to absurd conclusions, e.g., it's not worthwhile to create an infinite number of infinitely happy and well-off people if the cost is that your shoulder itches for a few seconds. It's at least a little worthwhile to create people with awesome lives, even if they should get weighted less than currently existent people.
Updating on someone else's decision to accept or reject a position should depend on their reason for their position. Information cascades is relevant.