At which point the humans running this NN will notice that it likes to go around risk control measures and will... persuade it that it's a bad idea.

How? By instituting more complex control measures? Then you're back to the problem Kaj mentioned above.

Yes, but this means that a lot of very rich people are very incorrect as to what is important for their wealth.

They know about the factors they can control. After all, those are the ones they actually focus on.

So? An opaque neural net would quickly learn how to get around trade size restrictions if given the proper motivations.

Consider that if it had been the opposite - IQ was more a personal benefit than a country benefit - we'd be explaining it as "obviously smart people benefit themselves at the expense of others".

Yes, it's called basing your beliefs on the evidence.