= 27d567bc2d48d9f40e9ccc20ea370b15)
lukeprog comments on Rationality Quotes September 2013 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (456)
It's not so much that I'm a better truth finder, it's that I've had the privilege of thinking through the issues as a core component of my full time job for the past two years, and people like Caplan only raise points that have been accounted for in my model for a long time. Also, I think the most productive way to resolve these debates is not to argue the meta-level issues about social epistemology, but to have the object-level debates about the facts at issue. So if Caplan replies to Carl's comment and my own, then we can continue the object-level debate, otherwise... the ball's in his court.
This doesn't appear to be accurate. E.g. Carl & Paul changed my mind about the probability of hard takeoff. And when have I said that some public figure agreeing with me made me more sure I'm right? See also my comments here.
If I mention a public figure agreeing with me, it's generally not because this plays a significant role in my own estimates, it's because other people think there's a stronger correlation between social status and correctness than I do.
Yes, but why Caplan did not see it fit to think about the issue for a significant time, and you did?
There's also the AI researchers who have had the privilege of thinking about relevant subjects for a very long time, education, and accomplishments which verify that their thinking adds up over time - and who are largely the actual source for the opinions held by the policy makers.
By the way, note that the usual method of rejection of wrong ideas, is not even coming up with wrong ideas in the first place, and general non-engagement of wrong ideas. This is because the space of wrong ideas is much larger than the space of correct ideas.
What I expect to see in the counter-factual world where the AI risk is a big problem, is that the proponents of the AI risk in that hypothetical world have far more impressive and far more relevant accomplishments and credentials.
The first problem with highly speculative topics is that great many arguments exist in favour of either opinion on a speculative topic. The second problem is that each such argument relies on a huge number of implicit or explicit assumptions that are likely to be violated due to their origin as random guesses. The third problem is that there is no expectation that the available arguments would be a representative sample of the arguments in general.
Hmm, I was under the impression that you weren't a big supporter of the hard takeoff to begin with.
Well, your confidence should be increased by the agreement; there's nothing wrong with that. The problem is when it is not balanced by the expected decrease by disagreement.
There are a great many differences in our world model, and I can't talk through them all with you.
Maybe we could just make some predictions? E.g. do you expect Stephen Hawking to hook up with FHI/CSER, or not? I think... oops, we can't use that one: he just did. (Note that this has negligible impact on my own estimates, despite him being perhaps the most famous and prestigious scientist in the world.)
Okay, well... If somebody takes a decent survey of mainstream AI people (not AGI people) about AGI timelines, do you expect the median estimate to be earlier or later than 2100? (Just kidding; I have inside information about some forthcoming surveys of this type... the median is significantly sooner than 2100.)
Okay, so... do you expect more or fewer prestigious scientists to take AI risk seriously 10 years from now? Do you expect Scott Aaronson and Peter Norvig, within 25 years, to change their minds about AI timelines, and concede that AI is fairly likely within 100 years (from now) rather than thinking that it's probably centuries or millennia away? Or maybe you can think of other predictions to make. Though coming up with crisp predictions is time-consuming.
Well, I too expect some form of something that we would call "AI", before 2100. I can even buy into some form of accelerating progress, albeit the progress would be accelerating before the "AI" due to the tools using relevant technologies, and would not have that sharp of a break. I even do agree that there is a certain level of risk involved in all the future progress including progress of the software.
I have a sense you misunderstood me. I picture this parallel world where legitimate, rational inferences about the AI risk exist, and where this risk is worth working at in 2013 and stands out among the other risks, as well as any other pre-requisites for making MIRI worthwhile hold. And in this imaginary world, I expect massively larger support than "Steven Hawkins hooked up with FHI" or what ever you are outlining here.
You do frequently lament that the AI risk is underfunded, under-supported, and there's under-awareness about it. In the hypothetical world, this is not the case and you can only lament that the rational spending should be 2 billions rather than 1 billion.
edit: and of course, my true rejection is that I do not actually see rational inferences leading there. The imaginary world stuff is just a side-note to explain how non-experts generally look at it.
edit2: and I have nothing against FHI's existence and their work. I don't think they are very useful, or address any actual safety issues which may arise, though, but with them I am fairly certain they aren't doing any harm either (Or at least, the possible harm would be very small). Promoting the idea that AI is possible within 100 years, however, is something that increases funding for AI all across the board.
Right, this just goes back to the same disagreement in our models I was trying to address earlier by making predictions. Let me try something else, then. Here are some relevant parts of my model:
Luke, why are you arguing with Dmytry?
The question should not be about "highly credentialed" people alone, but about how they fare compared to people who are rather very low "credentialed".
In particular, on your list, I expect people with fairly low credentials to fare much worse, especially at identification of the important issues as well as on rational thinking. Those combine multiplicatively, making it exceedingly unlikely - despite the greater numbers of the credential-less masses - that people who lead the work on an important issue would have low credentials.
What's EA? Effective altruism? If it's an existential risk, it kills everyone, selfishness suffices just fine.
Ohh, come on. That is in no way a demonstration that insurance companies in general follow faulty strategies, and especially is not a demonstration that you could do better.
Indeed.
A selfish person protecting against existential risk builds a bunker and stocks it with sixty years of foodstuffs. That doesn't exactly help much.
The quality of life in a bunker is really damn low. Not to mention that you presumably won't survive this particular risk in a bunker.
For what existential risks is this actually an effective strategy?
A global pandemic that kills everyone?
No doubt! I wasn't comparing highly credentialed people to low-credentialed people in general. I was comparing highly credentialed people to Bostrom, Yudkowsky, Shulman, etc.
But why exactly would you expect conventional researchers in AI and related technologies (also including provable software, as used in the aerospace industry, and a bunch of other topics), with credentials and/or accomplishments in said fields, to fare worse on that list's score?
Furthermore, with regards to the rationality, risks of mistake, and such... very little was done that can be checked for correctness in a clear cut way - most is of such nature that even when wrong it would not be possible to conclusively demonstrate it wrong. The few things that can be checked... look, when you write an article like this , discussing irrationality of Enrico Fermi, there's a substantial risk of appearing highly arrogant (and irrational) if you get the technical details wrong. It is a miniature version of AI risk problem - you need to understand the subject, and if you don't, there's negative consequences. It is much, much easier to not goof up in things like that, than AI direction.
As you guys are researching into actual AI technologies, the issue is that one should be able to deem your effort less of a risk. Mere "we are trying to avoid risk and we think they don't" can't do. The cost of a particularly bad friendly AI goof-up is a sadistic AI (to borrow the term from Omohundro). A sadistic AI can probably run far more tortured minds than a friendly AI can run minds, by a very huge factor, so the risk of a goof up must be quite a lot lower than anyone demonstrated.
BTW, I went back and numbered the items in my list so they're easier to refer to.
Because very few people in general, including credentialed AI people, satisfy (1), (2), (3), (5), (6), (7)†, (8), (10), and (12), but Bostrom, Yudkowsky and Shulman rather uncontroversially do satisfy those items. I also expect B/Y/S to outperform most credentialed experts on (4), (9), and (11), but I understand that's a subjective judgment call and it would take a long time for me to communicate my reasons.
† The AI risk part of 7, anyway. Obviously, AI people specifically know a lot about AI.
Edit: Also, I'll briefly mention that I haven't downvoted any of your comments in this conversation.
Ok, let's go over your list, for the AI people.
If EA is effective altruism, that's not relevant because one doesn't have to be an altruist to care about existential risks.
I expect them to be able to come up with that independently if it is a good idea.
I expect intelligent people to be able to foresee risks, especially when prompted by the cultural baggage (modern variations on the theme of Golem)
Well, that ought to imply some generally better ability to evaluate hard to calculate probabilities, which would imply that you guys should be able to make quite a bit of money.
The question is how well are they calibrated, not how much time they spent. You guys see miscalibration of famous people everywhere, even in Enrico Fermi.
Once again, how unbiased is what's important, not how much time spent on a very specific way to acquire an ability. I expect most accomplished people to have encountered far more feedback on being right / being wrong through their education and experience.
Doesn't apply to people in AI related professions.
The way to raise VoI is prior history of thinking about something else for a living, with impressive results.
Well, less credentialed people are just like this except they don't have a laboratory inside of which they are sane, that's usually why they are less credentialed in the first place.
Of your 3, I only weakly expect Bostrom to have learned the necessary fundamentals for actually applying Bayes theorem correctly in somewhat non-straightforward cases.
Yes, the basic formula is simple, but derivations are subtle and complex for non independent evidence or cases involving loops in the graph or all those other things...
It's like arguing that you are better equipped for a job at Weta Digital than any employee there because you know quantum electrodynamics (the fundamentals of light propagation), and they're using geometrical optics.
I expect many AI researchers to understand the relevant mathematics a lot, lot better than the 3 on your list.
And I expect credentialed people in general to have a good understanding of the variety of derivative tricks that are used to obtain effective results under uncertainty when the Bayes theorem can not be effectively applied.
Yeah, well, and I expect non-credentialed people to have too much to lose from backing out of it in the event that the studies return a negative.
You lose me here.
I would make a different list, anyway. There's my list:
Relevant expertise as measured by educational credentials and/or accomplishments. Expertise is required for correctly recognizing risks (e.g. an astronomer is better equipped for recognizing risks from the outer space, a physicist for recognizing faults in a nuclear power plant design, et cetera)
Proven ability to make correct inferences (largely required for 1).
Self preservation (most of us have it)
Lack of 1 is an automatic dis-qualifier in my list. It doesn't matter how much you are into things that you think are important for identifying, say, faults in a nuclear power plant design. If you are not an engineer, a physicist, or the like, you aren't going to qualify for that job via some list you make yourself, which conveniently omits (1).
edit: list copy paste failed.