If it's worth saying, but not worth its own post, then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should start on Monday, and end on Sunday.
4. Unflag the two options "Notify me of new top level comments on this article" and "
Too bad (or actually good) we can't actually see those superintelligent arguments. I wonder which direction they would take.
The author should perhaps describe them indirectly, i.e. not quote them (because the author is not a superintelligence, and cannot write superintelligent arguments), but describe reactions of other people after reading them. Those other people should generally become convinced about the validity of the arguments (because in-universe the arguments are superintelligent), but that can happen gradually, so in the initial phases they can be just generally impressed "hey, it actually makes more sense than I expected originally", and only after reading the whole document they would will become fully brainwashed (but perhaps not able to reproduce the argument in its full power, so they would urge the protagonist to read the original document). Random fragments of ideas can be thrown here and there, e.g. reported by people who read the superintelligent argument halfway. Perhaps the AI could quote Plato about how pure knowledge is the best knowledge (used as an excuse for why AI does not research something practical instead).
Thanks. In my imagination, the AI does some altruistic work, but spends most of its resources justifying the total expenditure. In that way, it would be similar to cults that do some charitable work, but spend most of their resources brainwashing people. But "rogue lawyer" is probably a better analogy than "cult guru" because the arguments are openly released. The AI develops models of human brain types in increasingly detailed resolutions, and then searches over attractive philosophies and language patterns, allowing it to accumulate c... (read more)