Z.M.: interesting discussion. weapons of math destruction is a wickedly clever phrase. Still, I can hope for more than "FAI must optimize something, we know not what. Before we can figure out what to optimize we have to understand Recursive Self Improvement. But we can't talk about that because it's too dangerous."

Nick: Yes, science is about models, as that post says. Formal models. It does not seem unreasonable to hope that some are forthcoming. Surely that is the goal. The post you reference is complaining about people making a distinction between the theoretical possibility of different levels of intelligence without any rational basis. That doesn't seem to be the same thing as merely asking for a little precision in the definitions of "intelligence", "self improvement", and "friendliness".