= 27d567bc2d48d9f40e9ccc20ea370b15)
multifoliaterose comments on The Importance of Self-Doubt - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (726)
I'm inclined to think that Eliezer's clear confidence in his own very high intelligence and his apparent high estimation of his expected importance (not the dictionary-definition "expected", but rather, measured as an expected quantity the usual way) are not actually unwarranted, and only violate the social taboo against admitting to thinking highly of one's own intelligence and potential impact on the world, but I hope he does take away from this a greater sense of the importance of a "the customer is always right" attitude in managing his image as a public-ish figure. Obviously the customer is not always right, but sometimes you have to act like they are if you want to get/keep them as your customer... justified or not, there seems to be something about this whole endeavour (including but not limited to Eliezer's writings) that makes people think !!!CRAZY!!! and !!!DOOMSDAY CULT!!!, and even if is really they who are the crazy ones, they are nevertheless the people who populate this crazy world we're trying to fix, and the solution can't always just be "read the sequences until you're rational enough to see why this makes sense".
I realize it's a balance; maybe this tone is good for attracting people who are already rational enough to see why this isn't crazy and why this tone has no bearing on the validity of the underlying arguments, like Eliezer's example of lecturing on rationality in a clown suit. Maybe the people who have a problem with it or are scared off by it are not the sort of people who would be willing or able to help much anyway. Maybe if someone is overly wary of associating with a low-status yet extremely important project, they do not really intuitively grasp its importance or have a strong enough inclination toward real altruism anyway. But reputation will still probably count for a lot toward what SIAI will eventually be able to accomplish. Maybe at the point of hearing and evaluating the arguments, seeming weird or high-self-regard-taboo-violating on the surface level will only screen off people who would not have made important contributions anyway, but it does affect who will get far enough to hear the arguments in the first place. In a world full of physics and math and AI cranks promising imminent world-changing discoveries, reasonably smart people do tend to build up intuitive nonsense-detectors, build up an automatic sense of who's not even worth listening to or engaging with; if we want more IQ 150+ people to get involved in existential risk reduction, then perhaps SIAI needs to make a greater point of seeming non-weird long enough for smart outsiders to switch from "save time by evaluating surface weirdness" mode to "take seriously and evaluate arguments directly" mode.
(Meanwhile, I'm glad Eliezer says "I have a policy of keeping my thoughts on Friendly AI to the object level, and not worrying about how important or unimportant that makes me", and I hope he takes that seriously. But unfortunately, it seems that any piece of writing with the implication "This project is very important, and this guy happens, through no fault of his own, to be one of very few people in the world working on it" will always be read by some people as "This guy thinks he's one of the most important people in the world". That's probably something that can't be changed without downplaying the importance of the project, and downplaying the importance of FAI probably increases existential risk enough that the PR hit of sounding overly self-important to probable non-contributors may be well worth it in the end.)
Leaving aside the question of whether such apparently strong estimation is warranted in the case at hand; I would suggest that there's a serious possibility that the social taboo that you allude to is adaptive; that having a very high opinion of oneself (even if justified) is (on account of the affect heuristic) conducive to seeing a halo around oneself, developing overconfidence bias, rejecting criticisms prematurely, etc. leading to undesirable epistemological skewing.
Same here.
It's easy to blunt this signal.
Suppose that any of:
A billionaire decided to devote most of his or her wealth to funding Friendly AI research.
A dozen brilliant academics became interested in and started doing Friendly AI research.
The probability of Friendly AI research leading to a Friendly AI is sufficiently low so that another existential risk reduction effort (e.g. pursuit of stable whole brain emulation) is many orders of magnitude more cost-effective at reducing existential risk than Friendly AI research.
Then the Eliezer would not (by most estimations) be the highest utilitarian expected value human in the world. If he were to mention such possibilities explicitly this would greatly mute the undesired connotations.
If I thought whole-brain emulation were far more effective I would be pushing whole-brain emulation, FOR THE LOVE OF SQUIRRELS!
Good to hear from you :-)
My understanding is that at present there's a great deal of uncertainty concerning how future advanced technologies are going to develop (I've gotten an impression that e.g. Nick Bostrom and Josh Tenenbaum hold this view). In view of such uncertainty, it's easy to imagine new data emerging over the next decades that makes it clear that pursuit of whole-brain emulation (or some currently unimagined strategy) is a far more effective strategy for existential risk reduction than Friendly AI research.
At present it looks to me like a positive singularity is substantially more likely to occur starting with whole-brain emulation than with Friendly AI.
Various people have suggested to me that initially pursuing Friendly AI might have higher expected value on the chance that it turns out to be easy. So I could imagine that it's rational for you personally to focus your efforts on Friendly AI research (EDIT: even if I'm correct in my estimation in the above point). My remarks in the grandparent above were not intended as a criticism of your strategy.
I would be interested in hearing more about your own thinking about the relative feasibility of Friendly AI vs. stable whole-brain emulation and current arbitrage opportunities for existential risk reduction, whether on or off the record.
That's an interesting claim, and you should post your analysis of it (e.g. the evidence and reasoning that you use to form the estimate that a positive singularity is "substantially more likely" given WBE).
There's a thread with some relevant points (both for and against) titled Hedging our Bets: The Case for Pursuing Whole Brain Emulation to Safeguard Humanity's Future. I hadn't looked at the comments until just now and still have to read them all; but see in particular a comment by Carl Shulman.
After reading all of the comments I'll think about whether I have something to add beyond them and get back to you.
You may want to read this paper I presented at FHI. Note that there's a big difference between the probability of risk conditional on WBE coming first or AI coming first and marginal impact of effort. In particular some of our uncertainty is about logical facts about the space of algorithms and technology landscape, and some of it is about the extent and effectiveness of activism/intervention.
Thanks for the very interesting reference! Is it linked on the SIAI research papers page? I didn't see it there.
I appreciate this point which you've made to me previously (and which appears in your comment that I linked above!).
Do you mean that the role of ems is in developing FAI faster (as opposed to biological-human-built FAI), or are you thinking of something else? If ems merely speed time up, they don't change the shape of FAI challenge much, unless (and to the extent that) we leverage them in a way we can't for the human society to reduce existential risk before FAI is complete (but this can turn out worse as well, ems can well launch the first arbitrary-goal AGI).
That's the main thing that's worried me about the possibility of ems coming first. But it depends on who is able to upload and who wants to, I suppose. If an average FAI researcher is more likely to upload, increase their speed, and possibly make copies of themselves than an average non-FAI AGI researcher, then it seems like that would be a reduction in risk.
I'm not sure whether that would be the case — a person working on FAI is likely to consider their work to be a matter of life and death, and would want all the speed increases they could get, but an AGI researcher may feel the same way about the threat to their career and status posed by the possibility of someone else getting to AGI first. And if uploading is very expensive at first, it'll only be the most well-funded AGI researchers (i.e. not SIAI and friends) who will have access to it early on and will be likely to attempt it (if it provides enough of a speed increase that they'd consider it to be worth it).
(I originally thought that uploading would be of little to no help in increasing one's own intelligence (in ways aside from thinking the same way but faster), since an emulation of a brain isn't automatically any more comprehensible than an actual brain, but now I can see a few ways it could help — the equivalent of any kind of brain surgery could be attempted quickly, freely, and reversibly, and the same could be said for experimenting with nootropic-type effects within the emulation. So it's possible that uploaded people would get somewhat smarter and not just faster. Of course, that's only soft self-improvement, nowhere near the ability to systematically change one's cognition at the algorithmic level, so I'm not worried about an upload bootstrapping itself to superintelligence (as some people apparently are). Which is good, since humans are not Friendly.)
There's a lot to respond to here. Some quick points:
It should be born in mind that greatly increased speed and memory may by themselves strongly affect a thinking entity. I imagine that if I could think a million times as fast I would think a lot more carefully about my interactions with the outside world than I do now.
I don't see any reason to think that SIAI will continue to be the only group thinking about safety considerations. If nothing else, SIAI or FHI can raise awareness of the dangers of AI within the community of AI researchers.
Assuming that brain uploads precede superhuman artificial intelligence, it would obviously be very desirable to have the right sort of human uploaded first.
I presently have a very dim view as to the prospects for modern day humans developing Friendly AI. This skepticism is the main reason why I think that pursuing whole-brain emulations first is more promising. See the comment by Carl that I mentioned in response to Vladimir Nesov's question. Of course, my attitude on this point is subject to change with incoming evidence.
Sped-up ems have slower computers relative to their thinking speed. If Moore's Law of Mad Science means that increasing computing power allows researchers to build AI with less understanding (and thus more risk of UFAI), then a speedup of researchers relative to computing speed makes it more likely that the first non-WBE AIs will be the result of a theory-intensive approach with high understanding. Anders Sandberg of FHI and I are working on a paper exploring some of these issues.
This argument lowers the estimate of danger, but AIs developed on relatively slow computers are not necessarily theory-intense, could also be coding-intense, which leads to UFAI. And theory-intense doesn't necessarily imply adequate concern about AI's preference.
My idea here is the same as the one that Carl Shulman mentioned in a response to one of your comments from nine months ago.