= 27d567bc2d48d9f40e9ccc20ea370b15)
Wei_Dai comments on Will the ems save us from the robots? - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (43)
Is the standard WBE analysis written up anywhere? By that phrase do you mean to include the "number of person-months" of work by FHI/SIAI that you mentioned earlier? I really am uncertain how far FHI/SIAI has pushed the analysis in these areas, and my questions were meant to be my attempt to figure that out. But it does seem like most of our disagreement is over decision theory rather than WBE, so let's move the focus there.
I also think that's most likely the case, but there's a significant chance that it isn't. I have not heard a strong argument why decision theory must be a very small piece of the AGI picture (and I did bring up this question on the decision theory mailing list), and in my state of ignorance it doesn't seem crazy to think that maybe with the right decision theory and just a few other key pieces of technology, AGI would be possible.
As for thinking there is ~0 chance of an FAI research program leading to safe AI, my reasoning is that with FAI we're dealing with seemingly impossible problems like ethics and consciousness, as well as numerous other philosophical problems that aren't quite thousand-years old, but still look quite hard. What are the chances all these problems get solved in a few decades, barring IA and WBE? If we do solve them, we still have to integrate the solutions into an AGI design, verify its correctness, avoid Friendliness-impacting implementation bugs, and do all of that before other AGI projects take off.
It's the social consequences that I'm most unsure about. It seems like if SIAI can keep "ownership" over the decision theory ideas and use it to preach AI risk, then that would be beneficial, but it could also be the case that the ideas take on a life of their own and we just end up having more people go into decision theory because they see it as a fruitful place to get interesting technical results.
On one hand, machine intelligence is all about making decisions in the face of uncertainty - so from this perspective, decision theory is central.
On the other hand, the basics of decision theory do not look that complicated - you just maximise expected utility. The problems seem to be mostly down to exactly how to do that efficiently.
The idea that safe machine intelligence will be assisted by modifications to decision theory to deal with "esoteric" corner cases seems to be mostly down to Eliezer Yudkowsky. I think it is a curious idea - but I am very happy that it isn't an idea that I am faced with promoting.
Isn't AIXI a counter-example to that? We could give it unlimited computing power, and it would still screw up badly, in large part due to a broken decision theory, right?
Kinda, yes. Any problem is a decision theory problem - in a sense. However, we can get a long way without the wirehead problem, utility counterfeiting, and machines mining their own brains causing problems.
From the perspective of ordinary development these don't look like urgent issues - we can work on them once we have smarter minds. We need not fear not solving them too much - since if we can't solve these problems our machines won't work and nobody will buy them. It would take security considerations to prioritise these problems at this stage.