= 27d567bc2d48d9f40e9ccc20ea370b15)
multifoliaterose comments on The Importance of Self-Doubt - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (726)
This post suffers from lumping together orthogonal issues and conclusions from them. Let's consider individually the following claims:
A priori, from (8) we can conclude (9). But assuming the a priori improbable (7), (8) is a rational thing for X to conclude, and (9) doesn't automatically follow. So, at this level of analysis, in deciding whether X is overconfident, we must necessarily evaluate (7). In most cases, (7) is obviously implausible, but the post itself suggests one pattern for recognizing when it isn't:
Thus, "doing something which very visibly and decisively alters the fate of humanity" is the kind of evidence that allows to conclude (7). But unfortunately there is no royal road to epistemic rationality, we can't require this particular argument that (7) in all cases. Sometimes the argument has an incompatible form.
In our case, the shape of the argument that (7) is as follows. Assuming (2), from (3) and (4) it follows that (5), and from (1), (5) and (6) we conclude (7). Note that the only claim about a person is (4), that their work contributes to development of FAI. All the other claims are about the world, not about the person.
Given the structure of this argument for the abhorrent (8), something being wrong with the person can only affect the truth of (4), and not of the other claims. In particular, the person is overconfident if person X's work doesn't in fact contribute to FAI (assuming it's possible to contribute to FAI).
Now, the extent of overconfidence in evaluating (4) is not related to the weight of importance conveyed by the object level conclusions (1), (2) and (3). One can be underconfident about (4) and still (8) will follow. In fact, (8) is rather insensitive to the strength of assertion (4): even if you contribute to FAI a little bit, but the other object level claims hold, your work is still very important.
Finally, my impression is that Eliezer is indeed overconfident about his ability to technically contribute to FAI (4), but not to the extent this post suggests, since as I said the strength of claim (8) has nothing to do with the level of overconfidence in (4), and even small contribution to FAI is enough to conclude (8) given other object level assumptions. Indeed, Eliezer never claims that success is assured:
On the other hand, only few people are currently in the position to claim (4) to any extent. One needs to (a) understand the problem statement, (b) be talented enough, and (c) take the problem seriously enough to direct serious effort at it.
My ulterior motive to elaborating this argument is to make the situation a little bit clearer to myself, since I claim the same role, just to a smaller extent. (One reason I don't have much confidence is that each time I "level up", last time around this May, I realize how misguided my past efforts were, and how much time and effort it will take to develop the skillset necessary for the next step.) I don't expect to solve the whole problem (and I don't expect Eliezer or Marcello or Wei to solve the whole problem), but I do expect that over the years, some measure of progress can be made by mine and their efforts, and I expect other people will turn up (thanks to Eliezer's work on communicating the problem statement of FAI and new SIAI's work on spreading the word) whose contributions will be more significant.
Your analysis is very careful and I agree with almost everything that you say.
I think that one should be hesitant to claim too much for a single person on account of the issue which Morendil raises - we are all connected. Your ability to work on FAI depends on the farmers who grow your food, the plumbers who ensure that you have access to running water, the teachers who you learned from, the people at Google who make it easier for you to access information, etc.
I believe that you (and others working on the FAI problem) can credibly hold the view that your work has higher expected value to humanity than that of a very large majority (e.g. 99.99%) of the population. Maybe higher.
I don't believe that Eliezer can credibly hold the view that he's the highest expected value human who has ever lived. Note that he has not offered a disclaimer denying the view that JRMayne has attributed to him despite the fact that I have suggested that he do so twice now.
You wrote elsewhere in the thread:
Does it mean that we need 10^9 Eliezer-level researchers to make progress? Considering that Eliezer is probably at about 1 in 10000 level of ability (if we forget about other factors that make research in FAI possible, such as getting in the frame of mind of understanding the problem and taking it seriously), we'd need about 1000 times more human beings than currently exists on the planet to produce a FAI, according to your estimate.
How does this claim coexist with the one you've made in the above comment?
It doesn't compute, there is an apparent inconsistency between these two claims. (I see some ways to mend it by charitable interpretation, but I'd rather you make the intended meaning explicit yourself.)
Agreed, and I like to imagine that he reads that and thinks to himself "only 10000? thanks a lot!" :)
In case anyone takes the above too seriously, I consider it splitting hairs to talk about how much beyond 1 in 10000 smart anyone is - eventually, motivation, luck, and aesthetic sense / rationality begin to dominate in determining results IMO.
No, in general p(n beings similar to A can do X) does not equal n multiplied by p(A can do X).
I'll explain my thinking on these matters later.
Yes, strictly speaking we'd need even more, if that. The more serious rendition of my remark is that you seem to imply that the problem itself is not solvable at all, by proxy of the estimate of Eliezer's ability to contribute to the solution. But it's OK, informal conclusions differ; what's not OK is that in the other comment you seem to contradict your claim.
Edit: I was not thinking clearly here.
No. There is a very small chance that I will be able to move my couch down the stairs alone. But it's fairly likely that I and my friend will be able to do it together.
Similarly, 10^5 Eliezer-level researchers would together constitute a research community that could do things that Eliezer himself has less than probability 10^(-5) of doing on his own.
Agreed, I was not thinking clearly. The original comment stands, since what you suggest is one way to dissolve the apparent inconsistency, but my elaboration was not lucid.
Tyrrel_MacAllister's remark is a significant part of what I have in mind.
I presently think that the benefits of a (modestly) large and diverse research community are very substantial and that SIAI should not attempt to research Friendly AI unilaterally but rather should attempt to collaborate with existing institutions.
I agree about the benefits of larger research community, although feasibility of "collaborating with existing institutions" is in question, due to the extreme difficulty of communicating the problem statement. There are also serious concerns about the end-game, where it will be relatively easy to instantiate a random-preference AGI on the basis of tools developed in the course of researching FAI.
Although the instinct is to say "Secrecy in science? Nonsense!", it would also be an example of outside view, where one completes a pattern while ignoring specific detail. Secrecy might make the development of a working theory less feasible, but if open research makes the risks of UFAI correspondingly even worse, it's not what we ought to do.
I'm currently ambivalent on this point, but it seems to me that at least preference theory (I'll likely have a post on that on my blog tomorrow) doesn't directly increase the danger, as it's about producing tools sufficient only to define Friendliness (aka human preference), akin to how logic allows to formalize open conjectures in number theory (of course, the definition of Friendliness has to reference some actual human beings, so it won't be simple when taken together with that, unlike conjectures in number theory), with such definition allowing to conclusively represent the correctness of any given (efficient algorithmic) solution, without constructing that solution.
On the other hand, I'm not confident that having a definition alone is not sufficient to launch the self-optimization process, given enough time and computing power, and thus published preference theory would constitute a "weapon of math destruction".
Hey, three days have passed and I want that post!
Maybe things could gradually change with more interface between people who are interested in FAI and researchers in academia.
I agree with this and believe that this could justify secrecy, but I think that it's very important that we hold the people who we trust with the end-game to very high standards for demonstrated epistemic rationality and scrupulousness.
I do not believe that the SIAI staff have met such standards. My belief on this matter regard is a major reason why I'm pursuing my current trajectory of postings.