= 27d567bc2d48d9f40e9ccc20ea370b15)
XiXiDu comments on A Brief Overview of Machine Ethics - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (90)
See, that one person who donated the current balance of his bank account got 52 upvotes for it. Now I'm not particularly shocked by him doing that or the upvotes. I don't worry that all that money might be better spend somehow. What drives me is curiosity mixed with my personality, I want to do what's right. That is the reason for why I criticize and why some comments may seem, or actually are derogatory. I think it needs to be said, I believe I can provoke feedback that way and learn more about the underlying rational. I desperately try to figure out if there is something I am missing.
I haven't read most of the sequences yet, let me explain why. I'm a really slow reader, I have almost no education and need a lot of time to learn anything. I did a lot of spot tests, reading various posts and came across people who read the sequences but haven't been able to conclude that they should stop doing anything except trying to earn money for the SIAI. My conclusion is that reading the sequences shouldn't be a priority right now but rather learning the mathematical basics, programming and reading various books. But I still try to spend some time here to see if that assessment might be wrong.
My current take on the whole issue is that the sequences do not provide much useful insights. I already know that by all that we know today AGI is possible and that it is unlikely that humans are the absolute limit when it comes to intelligence. I intuitively agree with the notion that AGI in its abstract form (intelligence as an algorithm) doesn't share our values if you do not deliberately 'tell' it to care. I see that one can outweigh even a low probability of risks from AI by assuming a future galactic civilization that is at stake. So what is my problem? I've written hundreds of comments about all kinds of problems I have with it, but maybe the biggest problem is a simple bias. I have an overwhelming gut feeling telling me that something is wrong with all this. I also do not trust my current ability to assess the situation to the extent that I would sacrifice other more compelling goals right now. And I am simply risk-adverse. I know that there is always either a best choice or all options are equal, no matter what uncertainty. Maybe everything is currently speaking in favor of the SIAI, but I'm not able to ignore my gut feeling right now. Trying to do so frequently makes me reluctant to do anything at all. Something is very wrong, I can't pinpoint what it is right now so I'm throwing everything I got at it to see if the facade crumbles. So far it did not crumble but neither have I received much reassuring feedback.
My recent comments have been made after a night of no sleep and being in a bad mood. I wouldn't have written them in that way on another day. I even messaged Eliezer yesterday telling him that he can edit/delete any of my submissions here that might be harmful without having to fear that I will protest and therefore cause more trouble. I don't care about myself much, but I care not to hurt others or cause damage. Sadly I often become reluctant, then I say 'fuck it' and just go ahead to write something because I was overwhelmed by all the possible implications and subsequently ignored them.
What is really confusing is that, taken at face value, the SIAI is working on the most important and most dangerous problem anyone will ever face. The SIAI is trying to take over the universe! Yet all I see in its followers is extreme scope insensitivity. How so? Because if you seriously believe that someone else believes that he is trying to take over the multiverse then you don't just trust him because he wrote a few posts about rationality and being honest. If the stakes are high, people do everything. Ask yourself, what difference would you expect to see if Dr. Evil would disguise as Eliezer Yudkowsky? Why wouldn't he write the sequences, why wouldn't he claim to be implementing CEV? That is one of the problems that make me feel that something is wrong here. Either people really don't believe all this stuff about fooming AI, galactic civilizations and the ability of the SIAI to create a seed AI, or I'm missing something. What I would expect to see is people asking for transparency. I expect people to demand oversight and ask how exactly their money is being spend. I expect people to be much more critical and to not just believe Yudkowsky but ask for data and progress reports. Nada.
It's worth noting that AGI is decades away; no one's trying to take over the universe just yet. In this light, donations to SingInst now are better seen as funding preliminary research and outreach regarding this important problem, rather than funding AI construction.
What sort of data and progress reports are you looking for? Glancing at the first two pages of the SingInst blog, I see a list of 2010 publications, and newsletters for last July and October. There's certainly room for criticism (e.g., "Why no newsletter since last October?" or "All this outreach is not very useful; I want to see incremental progress towards FAI"), but I wouldn't say there've been no progress reports.
As far as I know Yudkowsky is currently writing a book. He earnt $95,550 last year.
What I can't reconcile right now is the strong commitment and what is actually being done. Quite a few people here actually seem to donate considerable amounts of their income to the SIAI. No doubt writing the sequences, a book and creating an online community is pretty cool but does not seem to be too cost intensive. At least others manage to do that without lots of people sending them money. I myself donated 3 times. But why are many people acting like the SIAI is the only charity that currently deserves funding, why is nobody asking if they actually need more money or if they are maybe sustainable right now? I haven't heard anything about the acquisition of a supercomputer, field experiments in neuroscience or the hiring of mathematicians. All that would justify further donations. I feel people here are not critical and demanding enough.
Upvoting for honesty and posting a true rejection.
Even if you're a slow reader, I think that it is very, very worth it to read most of the sequences. I've not read QM, Evolution, Decision Theory, and parts of Metaethics/ Human's guide to words, but I think that reading the others has drastically increased my rationality (especially the Core Sequences.) I don't think that reading technical books would have done so nearly as much because I find reading prose much more engaging than math.
I've recently concluded that I should place a 'highly suspect' marker on my thoughts (especially negative generalizations) if I am very hungry or tired. I tend to be quite irritable in both cases -- I'll get into arguments in which I'm really not interested in finding truth, but just getting a high from bashing the other person into the ground (please note that I am sharing my own experiences, not accusing you of this.) You may want to type these comments out so that you don't lose the thought but wait to post them until you're feeling better.
I've had these same thoughts before and since resolved them, but I've run out of mental steam and need to do some schoolwork. I may edit this or make a separate reply to this later.
Edit: Bolded script in this post was added for clarification -- bolding does not indicate emphasis here.
Interesting thought, I'll admit hadn't actually considered that (I have a general problem with being too trusting and not seeing ulterior motives, although I suspect most people really aren't very dishonest).
I can see a few reasons why others might not be asking:
1) Its unlikely to get an answer. There hasn't been a whole lot of willingness to respond to similar requests in the past, EY has a thing about not giving in to demands. This doesn't really explain why people are still donating.
2) The number of genuine Dr Evils in the world is very small. Historically the most dangerous individuals have been the well-intentioned but deluded rather than the rationally malicious, which is odd since the latter category seem much more dangerous and therefore provides evidence of their rarity. Maybe people are just making an expected utility calculation and determining that the Dr Evil hypothesis is unlikely enough to trust SIAI anyway.
3) Eliezer is not the whole of SIAI, he is not even in charge. Some of the people involved have existing track records, if there is a conspiracy it runs very deep. I suppose its possible he has tricked every other member of the organization, but we are now adding a lot of burdensome details to what was already a fairly unlikely hypothesis.
4) If there are any real Dr Evils out there, then SIAI transparency might actually help them by giving away SIAI ideas while Dr Evil keeps his ideas to himself and as a result finishes his design first.
5) If I was Dr Evil trying to build an AI, then I wouldn't say that was what I was doing, since AI is quite a hard sell and will only get donations from a limited demographic (even more so for an out-of-the-mainstream idea like FAI). I would found the "organization for the protection of puppies kittens and bunnies" or something like that, which will probably get more donation money (or maybe even go into business rather than charity, since current evidence suggests that is overwhelmingly the most effective way to make large amounts of money).
Frankly, rather than a Dr Evil who wants to take over the Galaxy (I don't think he's ever said anything about the multiverse) a much more likely prospect is a conman who's found an underused niche. Of course, this wouldn't explain how he got some fairly big names like Jaan Tallinn and Edwin Edward to sponsor his donation drive.
Most of these reasons are being quite charitable to LW members, and unfortunately I suspect my own reason is the most common.
Yes, it is impossible to distinguish a sincere optimist from a perfectly selfish sociopath. At least until they gain power (or move to an audience where the signalling game is played at a higher level of sophistication than that of conveying altruism).
In that case, I would expect a stupid Eliezer Yudkowsky. But one shouldn't actually reason this way, the question is, what do you anticipate, given observations actually made; not how plausible are the observations actually made, given an uncaused hypothesis.
You can't compute P(H|E) without computing P(E|H).
But one shouldn't confuse the two.
Why is evil stupid and what evidence is there that Yudkowsky is smart enough not to be evil?
If you got someone working on friendly AI you better ask if the person is friendly in the first place. You also shouldn't make conclusions based on the output of the subject of your conclusions. If Yudkowsky states what is right and states that he will do what is right that provides no evidence about the rightness and honesty of those statements. Besides, the most advanced statements about Yudkowsky's intentions are CEV and the meta-ethics sequence. Both are either criticized or not understood.
The question should be, what is the worst-case scenario regarding Yudkowsky and the SIAI and how can we discern it from what he is signaling? If the answer isn't clear, one should ask for transparency and oversight.
You seem to be under the impression that Eliezer is going to create an artificial general intelligence, and oversight is necessary to ensure that he doesn't create one which places his goals over humanity's interests. It is important, you say, that he is not allowed unchecked power. This is all fine, except for one very important fact that you've missed.
Eliezer Yudkowsky can't program. He's never published a nontrivial piece of software, and doesn't spend time coding. In the one way that matters, he's a muggle. Ineligible to write an AI. Eliezer has not positioned himself to be the hero, the one who writes the AI or implements its utility function. The hero, if there is to be one, has not yet appeared on stage. No, Eliezer has positioned himself to be the mysterious old wizard - to lay out a path, and let someone else follow it. You want there to be oversight over Eliezer, and Eliezer wants to be the oversight over someone else to be determined.
But maybe we shouldn't trust Eliezer to be the mysterious old wizard, either. If the hero/AI programmer comes to him with a seed AI, then he knows it exists, and finding out that a seed AI exists before it launches is the hardest part of any plan to steal it and rewrite its utility function to conquer the universe. That would be pretty evil, but would "transparency and oversight" make things turn out better, or worse? As far as I can tell, transparency would mean announcing the existence of a pre-launch AI to the world. This wouldn't stop Eliezer from make a play to conquer the universe, but it would present that option to everybody else, including at least some people and organizations who are definitely evil.
So that's a bad plan. A better plan would be to write a seed AI yourself, keep it secret from Eliezer, and when it's time to launch, ask for my input instead.
(For the record: I've programmed in C++, Python, Java, wrote some BASIC programs on a ZX80 when I was 5 or 6, and once very briefly when MacOS System 6 required it I wrote several lines of a program in 68K assembly. I admit I haven't done much coding recently, due to other comparative advantages beating that one out.)
I can't find it by search, but haven't you stated that you've written hundreds of KLOC?
Yep, he have.
Sounds about right. It wasn't good code, I was young and working alone. Though it's more like the code was strategically stupid than locally poorly written.
I disagree based on the following evidence:
You further write:
I'm not aware of any reason to believe that recursively self-improving artificial general intelligence is going to be something you can 'run away with'. It looks like some people here think so, that there will be some kind of, with hindsight, simple algorithm for intelligence that people can just run and get superhuman intelligence. Indeed, transparency could be very dangerous in that case. But that doesn't mean it is an all or nothing decision. There are many other reasons for transparency, including reassurance and the ability to discern a trickster or impotent individual from someone who deserves more money. But as I said, I don't see that anyway. It'll more likely be a blue sheet of different achievements that are each not dangerous on their own. I further think it will be not just a software solution but also a conceptual and computational revolution. In those cases an open approach will allow public oversight. And even if someone is going to run with it, you want them to use your solution rather than one that will most certainly be unfriendly.
Evil is not necessarily stupid (well, it is, if we are talking about humans, but let's abstract from that). Still, it would take a stupid Dr Evil to decide that pretending to be Eliezer Yudkowsky is the best available course of action.
You don't think that being Eliezer Yudkowsky is an effective way to accomplish the task at hand? What should Dr Evil do, then?
FWIW, my usual comparison is not with Dr Evil, but with Gollum. The Singularity Instutute have explicitly stated said they are trying to form "The Fellowship of the AI". Obviously we want to avoid Gollum's final scene.
Gollum actually started out good - it was the exposure to the ring that caused problems later on.
I seem to remember Smeagol being an unpleasant chap even before Deagol found the ring. But admittedly, we weren't given much.
Transparency is listed as being desirable here:
However, apparently, this doesn't seem to mean open source software - e.g. here:
You equivocate two unrelated senses of "transparency".