Hm, that is a better point, it seems then most of my objections are just to the wording. Most intelligent people are also shy etc. and that is why they end up being math researchers instead of being Steve Jobs. If an intelligent person could edit in his mind courage, dedication, charme... that would be powerful.
But I think self-modification would be powerful even without very high IQ, 120 would already make one pretty succesful.
Or is it more IQ being necessary for efficient self-modification?
My point is, this sounds like a powerful combination, but probably not the intelligence explosion kind.
The caste stuff: really elegant steelmanning, congrats. But I think kind of missing the point, probably I explained myself wrong. Basically an IQ-meritocracy requires a market based system, exchanged based one, where what you get is very roughly proportional to what you give to others. However most of the planet is not exchange based but power based. This is where intelligence is less useful. Imagine trying to compete with Stalin for the job of being Lenin's successor. What traits you need for it? First of all, mountains of courage, that guys is scary. Of course if you can self-edit, that is indeed extremely helpful in it... I did not factor that in. But broadly speaking, you don't just outsmart him. Power requires other traits. And of course it can very well be that you don't want power, you want to be a researcher... but in that situation you are forced to take orders from him so you may still want to topple the big boss or something.
Now of course if we see intelligence as simply the amount of sophistication put into self-editing, so seeing a higher intelligence as something that can self-edit e.g. charisma better than a lower intelligence... then these possibilities are indeed there. I am just saying, still no intelligence explosion, more like everything explosion, or maybe everything else but intelligence explosion. Charisma explosion, and so on... but I do agree that this combined can be very powerful.
Or is it more IQ being necessary for efficient self-modification?
Sounds like false dilemma if IQ is one of those things that can be modified. :D
To unpack the word, IQ approximately means how complex concepts can you "juggle" in your head. Without enough IQ, even if you had an easy computer interface to modify your own brain, you wouldn't understand what exactly you should do to achieve your goals (because you wouldn't sufficiently understand the concepts and the possible consequences of the changes). That means, you would be making those chang...
Edit: Some people have misunderstood my intentions here. I do not in any way expect this to be the NEXT GREAT IDEA. I just couldn't see anything wrong with this, which almost certainly meant there were gaps in my knowledge. I thought the fastest way to see where I went wrong would be to post my idea here and see what people say. I apologise for any confusion I caused. I'll try to be more clear next time.
(I really can't think of any major problems in this, so I'd be very grateful if you guys could tell me what I've done wrong).
So, a while back I was listening to a discussion about the difficulty of making an FAI. One of the ways that was suggested to circumvent this was to go down the route of programming an AGI to solve FAI. Someone else pointed out the problems with this. Amongst other things one would have no idea what the AI will do in pursuit of its primary goal. Furthermore, it would already be a monumental task to program an AI whose primary goal is to solve the FAI problem; doing this is still easier than solving FAI, I should think.
So, I started to think about this for a little while, and I thought 'how could you make this safer?' Well, first of, you don't want an AI who completely outclasses humanity in terms of intellect. If things went Wrong, you'd have little chance of stopping it. So, you want to limit the AI's intellect to genius level, so if something did go Wrong, then the AI would not be unstoppable. It may do quite a bit of damage, but a large group of intelligent people with a lot of resources on their hands could stop it.
Therefore, what must be done is that the AI cannot modify parts of its source code. You must try and stop an intelligence explosion from taking off. So, limited access to its source code, and a limit on how much computing power it can have on hand. This is problematic though, because the AI would not be able to solve FAI very quickly. After all, we have a few genius level people trying to solve FAI, and they're struggling with it, so why should a genius level computer do any better. Well, an AI would have fewer biases, and could accumulate much more expertise relevant to the task at hand. It would be about as capable as solving FAI as the most capable human could possibly be; perhaps even more so. Essentially, you'd get someone like Turing, Von Neumann, Newton and others all rolled into one working on FAI.
But, there's still another problem. The AI, if left for 20 years working on FAI for 20 years let's say, would have accumulated enough skills that it would be able to cause major problems if something went wrong. Sure, it would be as intelligent as Newton, but it would be far more skilled. Humanity fighting against it would be like sending a young Miyamoto Musashi against his future self at his zenith i.e. completely one sided.
What must be done then, is the AI must have a time limit of a few years (or less) and after that time is past, it is put to sleep. We look at what it accomplished, see what worked and what didn't, and boot up a fresh version of the AI with any required modifications, and tell it what the old AI did. Repeat the process for a few years, and we should end up with FAI solved.
After that, we just make an FAI, and wake up the originals, since there's no point in killing them off at this point.
But there are still some problems. One, time. Why try this when we could solve FAI ourselves? Well, I would only try and implement something like this if it is clear that AGI will be solved before FAI is. A backup plan if you will. Second, what If FAI is just too much for people at our current level? Sure, we have guys who are one in ten thousand and better working on this, but what if we need someone who's one in a hundred billion? Someone who represents the peak of human ability? We shouldn't just wait around for them, since some idiot would probably just make an AGI thinking it would love us all anyway.
So, what do you guys think? As a plan, is this reasonable? Or have I just overlooked something completely obvious? I'm not saying that this would by easy in anyway, but it would be easier than solving FAI.