I don't yet know enough about AI to even attempt to answer that, I am just trying to form a position on intelligence itself, human intelligence. I don't think IQ is predictive of how much power people have. I don't think the world is an IQ meritocracy. I understand how it can look like one from the Silicon Valley, because the SV is an IQ meritocracy, where people who actually use intelligence to change the world go to, but in general the world is not so. IQ is more of a puzzle-solving ability and I think it transforms to world-changing power only when the bottleneck is specifically the lack of puzzle-solving ability. When we have infinite amounts of dedication, warm bodies, electricity and money to throw on a problem and the only thing missing is that we don't know how, then yes, smart people are useful, they can figure that out. But that is not the usual case. Imagine you are a 220 IQ Russian and just solved cold fusion and offer it to Putin out of patriotism. You probably get shot and the blueprints buried because it threatens the energy exports so important for their economy and for the power of his oligarch supporters. This is IMHO how intelligence works, if there is everything there, especially the will, to change the world, and only the know-how is missing, then it is useful, but it is not the typical case, and in all the other cases it does not help much. Yes, of course a superintelligence could figure out, say, nanotechnology, but why should we assume that main reason why the world is not a utopia is the lack of such know-how?
This is why I am so suspicious about it. That I am afraid the whole thing is Silicon Valley culture written large, and this is not predictive of how the world works. SV is full of people eager to change the world, have the dedication, the money, all is missing is knowing how. A superintelligence could help them, but it does not mean intelligence, or superintelligence, is a general power, a general optimization ability, a general goal-achievement ability. I think it is only true in the special cases when achieving goals requires solving puzzles. Usually, it is not unsolved puzzles that are standing between you and your goal.
Imagine I wanted to be dictator of a first-world country. Is there any puzzle I could solve if I had a gazillion IQ that could achieve that? No. I could read all the books about how to manipulate people and figure out the best things to say and the result would still be that people don't want to give up their freedom especially not to some uncharismatic fat nerds no matter how excellent sounding things he says. But maybe if it was an economic depression and I would be simply highly charismatic, ruthless, and have all the right ex-classmates... I would have a chance to pull that off without any puzzle-solving, just being smart enough to not sabotage myself with blunders, say, IQ 120 would do it.
And I am worried I am missing something huge. MIRI consists of far smarter people than I am, so I am almost certainly wrong... unless they have fallen in love with smartness so much that it created a pro-smart bias in them, unless it made them underestimate in how many cases efficient world changing has nothing to do with puzzle solving and has everything to do with beating obstacles with a huge hammer, or forging consensus or a million other things.
But I think the actual data about what 180+ IQ people are doing is on my side here. What is Kasparov doing? What is James Woods doing? Certainly not radically transforming the world. Nor taking it over.
I don't think IQ is predictive of how much power people have. I don't think the world is an IQ meritocracy.
Predictive isn't a binary catergory. Statistically IQ is predictive of a lot of things including higher social skills and lifespan.
...Imagine I wanted to be dictator of a first-world country. Is there any puzzle I could solve if I had a gazillion IQ that could achieve that? No. I could read all the books about how to manipulate people and figure out the best things to say and the result would still be that people don't want to give up their freedom
Edit: Some people have misunderstood my intentions here. I do not in any way expect this to be the NEXT GREAT IDEA. I just couldn't see anything wrong with this, which almost certainly meant there were gaps in my knowledge. I thought the fastest way to see where I went wrong would be to post my idea here and see what people say. I apologise for any confusion I caused. I'll try to be more clear next time.
(I really can't think of any major problems in this, so I'd be very grateful if you guys could tell me what I've done wrong).
So, a while back I was listening to a discussion about the difficulty of making an FAI. One of the ways that was suggested to circumvent this was to go down the route of programming an AGI to solve FAI. Someone else pointed out the problems with this. Amongst other things one would have no idea what the AI will do in pursuit of its primary goal. Furthermore, it would already be a monumental task to program an AI whose primary goal is to solve the FAI problem; doing this is still easier than solving FAI, I should think.
So, I started to think about this for a little while, and I thought 'how could you make this safer?' Well, first of, you don't want an AI who completely outclasses humanity in terms of intellect. If things went Wrong, you'd have little chance of stopping it. So, you want to limit the AI's intellect to genius level, so if something did go Wrong, then the AI would not be unstoppable. It may do quite a bit of damage, but a large group of intelligent people with a lot of resources on their hands could stop it.
Therefore, what must be done is that the AI cannot modify parts of its source code. You must try and stop an intelligence explosion from taking off. So, limited access to its source code, and a limit on how much computing power it can have on hand. This is problematic though, because the AI would not be able to solve FAI very quickly. After all, we have a few genius level people trying to solve FAI, and they're struggling with it, so why should a genius level computer do any better. Well, an AI would have fewer biases, and could accumulate much more expertise relevant to the task at hand. It would be about as capable as solving FAI as the most capable human could possibly be; perhaps even more so. Essentially, you'd get someone like Turing, Von Neumann, Newton and others all rolled into one working on FAI.
But, there's still another problem. The AI, if left for 20 years working on FAI for 20 years let's say, would have accumulated enough skills that it would be able to cause major problems if something went wrong. Sure, it would be as intelligent as Newton, but it would be far more skilled. Humanity fighting against it would be like sending a young Miyamoto Musashi against his future self at his zenith i.e. completely one sided.
What must be done then, is the AI must have a time limit of a few years (or less) and after that time is past, it is put to sleep. We look at what it accomplished, see what worked and what didn't, and boot up a fresh version of the AI with any required modifications, and tell it what the old AI did. Repeat the process for a few years, and we should end up with FAI solved.
After that, we just make an FAI, and wake up the originals, since there's no point in killing them off at this point.
But there are still some problems. One, time. Why try this when we could solve FAI ourselves? Well, I would only try and implement something like this if it is clear that AGI will be solved before FAI is. A backup plan if you will. Second, what If FAI is just too much for people at our current level? Sure, we have guys who are one in ten thousand and better working on this, but what if we need someone who's one in a hundred billion? Someone who represents the peak of human ability? We shouldn't just wait around for them, since some idiot would probably just make an AGI thinking it would love us all anyway.
So, what do you guys think? As a plan, is this reasonable? Or have I just overlooked something completely obvious? I'm not saying that this would by easy in anyway, but it would be easier than solving FAI.