It's that I can't imagine this game invoking any negative emotions stronger than sad novels and movies.
What's surprising is that Tuxedage seems to be actually hurt by this process, and that s/he seems to actually fear mentally damaging the other party.
In our daily lives we don't usually* censor emotionally volatile content in the fear that it might harm the population. The fact that Tuxedage seems to be more ethically apprehensive about this than s/he might about, say, writing a sad novel, is what is surprising.
I don't think s/he would show this level of apprehension about, say, making someone sit through Grave of the Firefles. If s/he can actually invoke emotions more intense than that through text only terminals to a stranger, then whatever s/he is doing is almost art.
Some people fall in love over text. What's so surprising?
That's real-world, where you can tell someone you'll visit them and there is a chance of real-world consequence. This is explicitly negotiated pretend play in which no real-world promises are allowed.
given how common mental illness is.
I...suppose? I imagine you'd have to have a specific brand of emotional volatility combined with immense suggestibility for this sort of thing to actually damage you. You'd have to be the sort of person who can be hypnotized against their will to do and feel things they actually don't want to do and feel.
At least, that's what I imagine. My imagination apparently sucks.
I'd guess that Tuxedage is hurt the same as the gatekeeper is because he has to imagine whatever horrors he inflicts on his opponent. Doing so causes at least part of that pain (and empathy or whatever emotion is at work) in him too. He has the easier part because he uses it as a tool and his mind has one extra layer of story-telling where he can tell himself "it's all a story". But part of 'that' story is winning and if he doesn't win part of these horrors fall back to him.
AI Box Experiment Update #3
Tuxedage (AI) vs Alexei (GK) - Gatekeeper Victory
Tuxedage (AI) vs Anonymous (GK) - Gatekeeper Victory
I have won a second game of AI box against a gatekeeper who wished to remain Anonymous.
This puts my AI Box Experiment record at 3 wins and 3 losses.