I don't agree that catgirls in that sense are orthogonal to porn, though. At all.

Parsing error, sorry. I meant that, since they'd been disclaimed, what was actually being talked about was orthogonal to porn.

No part, but you can't merely 'satisfy preferences'.. you have to also not-satisfy preferences that have a stagnating effect.

Only if you prefer to not stagnate (to use your rather loaded word :)

I'm not sure at what level to argue with you at... sure, I can simultaneously contain a preference to get fit, and a preference to play video games at all times, and in order to indulge A, I have to work out a system to suppress B. And it's possible that I might not have A, and yet contain other preferences C that, given outside help, would cause A to be added to my preference pool: "Hey dude, you want to live a long time, right? You know exercising will help with that."

All cool. But there has to actually be such a C there in the first place, such that you can pull the levers on it by making me aware of new facts. You don't just get to add one in.

for example, humans have an unhealthy, unrealistic, and excessive desire for certainty.

I'm not sure this is actually true. We like safety because duh, and we like closure because mental garbage collection. They aren't quite the same thing.

There's one particular character that likes fucking and killing.. and drinking.. and that's basically his main preferences. CelestAI satisfies those preferences, and that satisfaction can be considered as harm to him as a person.

(assuming you're talking about Lars?) Sorry, I can't read this as anything other than "he is aesthetically displeasing and I want him fixed".

Lars was not conflicted. Lars wasn't wishing to become a great artist or enlightened monk, nor (IIRC) was he wishing that he wished for those things. Lars had some leftover preferences that had become impossible of fulfilment, and eventually he did the smart thing and had them lopped off.

You, being a human used to dealing with other humans in conditions of universal ignorance, want to do things like say "hey dude, have you heard this music/gone skiing/discovered the ineffable bliss of carving chair legs"? Or maybe even "you lazy ass, be socially shamed that you are doing the same thing all the time!" in case that shakes something loose. Poke, poke, see if any stimulation makes a new preference drop out of the sticky reflection cogwheels.

But by the specification of the story, CelestAI knows all that. There is no true fact she can tell Lars that will cause him to lawfully develop a new preference. Lars is bounded. The best she can do is create a slightly smaller Lars that's happier.

Unless you actually understood the situation in the story differently to me?

Yes, but not in-itself. It needs to have a function in developing us as persons, which it will lose if it merely satisfies us.

I disagree. There is no moral duty to be indefinitely upgradeable.