I am interested in anything that allows better reasoning about these topics.
Mathematics has a somewhat limited use when discussing the orthogonality thesis. AIXI, and some calculations about the strength of optimisation processes and stuff like that. But when answering the question "is it likely that humans will build AIs with certain types of goals", we need to look beyond mathematics.
I won't pretend the argument in this post is strong - it's just, to use the technical term, "kinda neat" and I'd never seen it presented this way before.
What would you consider reasonable reasoning on questions like the orthogonality thesis in practice?
That's how religions were created, you know - they could not actually answer why lightning is thundering, why sun is moving through the sky, etc. So they did look way 'beyond' the non-faulty reasoning, in search for answers now (being inpatient), and got answers that were much much worse than no answers at all. I feel LW is doing precisely same thing with AIs. Ultimately, when you can't compute the right answer in the given time, you will either have no answer or compute a wrong one.
On the orthogonality thesis, it is the case that you can't answer this qu...
Just a minor thought connected with the orthogonality thesis: if you claim that any superintelligence will inevitably converge to some true code of morality, then you are also claiming that no measures can be taken by its creators to prevent this convergence. In other words, the superintelligence will be uncontrollable.