capacity for emotional healing
an AI (even if not in a society of AIs) could either have mistakes built into its structure or make mistakes when changing itself.
I am not very well versed in AI at all. But reading this, my automatic response is to question how an emotional response is different from any other response, for an AI.
I understand that emotional responses are different in complexity than trivial responses. But I think of emotional responses (for an AI) as fitting somewhere in the fairly straightforward continuum between "what color should my desktop be" and "how do I judge the validity of a moral structure to apply to humans when they cannot agree on any meaningful criteria themselves". I would assume that even AI emotional ecology is closer to the complex side of the spectrum, but it seems like a problem that should be fully open to internal inspection and modification by the AI -- or if it is limited, at least no more difficult to adjust than any equally important calculation.
Building an AI with hidden subconscious seems like an unfortunate combination of stupid and malicious. The most likely reason for such a thing to exist that I can think of is as a hidden backdoor to allow humans to manipulate the AI without it knowing what is going on, but inducing schizophrenia-like symptoms is probably not the sane way to control our constructs.
But I may be under-applying important concepts -- particularly, I may be underestimating the importance of emergent properties, especially in a hard takeoff scenario.
I brought up emotional healing because I'd recently read a strong example of it, but you raise a bunch of interesting points, and because, as I said, people seem to have a base state of emotional health-- a system 1 which is compatible with living well. It seems as though people have some capacity for improving their system 1 reactions, though it tends to be slow and difficult.
Let's see if I can generalize healing for AIs. I'm inclined to think that AIs will have something like a system 1 / system 2 distinction-- subsystems for faster/lower cost reactions ...
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.